Llama Corp

llamacorp

Welcome to this ”to-the-point” tutorial on how to quantize any Large Language Model (LLM) available on Hugging Face using llama.cpp. Whether you're a data scientist, a machine learning engineer, or simply an AI enthusiast, this guide is designed to clarify the process of model quantization and make it easy. By the end of this tutorial, you'll have a clear understanding of how to efficiently compress LLMs without significant loss in performance, enabling their deployment on resource-constrained environments. You can also use these model on your fav local setups using Ollama! GitGub Repo is here: https://github.com/mickymultani/QuantizeLLMs What is Model Quantization? Before we dive into the technicalities, let's briefly discuss what model quantization is and why it's important. Model quantization is a technique used to reduce the precision of the numbers used in a model's weights and activations. This process significantly reduces the model size and speeds up inference times, making it possible to deploy state-of-the-art models on devices with limited memory and computational power, such as mobile phones and embedded systems. Introducing llama.cpp llama.cpp is a powerful tool that facilitates the quantization of LLMs. It supports various quantization methods, making it highly versatile for different use cases. The tool is designed to work seamlessly with models from the Hugging Face Hub, which hosts a wide range of pre-trained models across various languages and domains. Setting Up Your Workspace This tutorial is designed with flexibility in mind, catering to both cloud-based environments and local setups. I personally conducted these steps on Google Colab, utilizing the NVIDIA Tesla T4 GPU, which provides a robust platform for model quantization and testing. However, you should not feel limited to this setup. The beauty of llama.cpp and the techniques covered in this guide is their adaptability to various environments, including local machines with GPU support. For instance, if you're using a MacBook with Apple Silicon, you can follow along and leverage the GPU support for model quantization, thanks to the cross-platform compatibility of the tools and libraries we are using. Setting Up on Google Colab Google Colab provides a convenient, cloud-based environment with access to powerful GPUs like the T4. If you choose Colab for this tutorial, make sure to select a GPU runtime by going to . This ensures that your notebook has access to the necessary computational resources. Runtime > Change runtime type > T4 GPU Running on MacBook with Apple Silicon For those opting to run the quantization process on a MacBook with Apple Silicon, ensure that you have the necessary development tools and libraries installed. While the setup might slightly differ from the Linux-based Colab environment, Python's ecosystem and the compilation of llama.cpp with CUDA support (or the equivalent for your specific GPU architecture) are well-documented, ensuring a smooth setup process. Setting Up Hugging Face Authentication Regardless of your platform, access to models from the Hugging Face Hub requires authentication. To seamlessly integrate this in your workflow, especially when working in collaborative or cloud-based environments like Google Colab, it's advisable to set up your Hugging Face authentication token securely. On Google Colab, you can safely store your Hugging Face token by using Colab's "Secrets" feature. This can be done by clicking on the "Key" icon in the sidebar, selecting "Secrets", and adding a new secret with the name and your Hugging Face token as the value. This method ensures that your token remains secure and is not exposed in your notebook's code. HF_TOKEN For local setups, consider setting the environment variable in your shell or utilizing the Hugging Face CLI to log in, thereby ensuring that your scripts have the necessary permissions to download and upload models to the Hub without hard-coding your credentials. HF_TOKEN Step-by-Step Tutorial Now, let's walk through the process of quantizing a model using llama.cpp. For this tutorial, we'll quantize the " " model from Hugging Face, but the steps can be applied to any model of your choice. google/gemma-2b-it 1. Setting Up Your Environment First, we need to clone the llama.cpp repository and install the necessary requirements: !git clone https://github.com/ggerganov/llama.cpp
!cd llama.cpp && LLAMA_CUBLAS=1 make && pip install -r requirements.txt This command clones the llama.cpp repository and compiles the necessary binaries with CUDA support for GPU acceleration. It also installs Python dependencies required for the process. 2. Downloading the Model Next, we download the model from Hugging Face Hub using the function. This function ensures that we have a local copy of the model for quantization: snapshot_download from huggingface_hub import snapshot_download

model_name = "google/gemma-2b-it"
base_model = "./original_model/"
snapshot_download(repo_id=model_name, local_dir=base_model, local_dir_use_symlinks=False) 3. Preparing the Model for Quantization Before quantizing, we convert the downloaded model to a format compatible with llama.cpp ( ). This step involves specifying the desired precision ( for half-precision floating point) and the output file: gguf f16 !mkdir ./quantized_model/
!python llama.cpp/convert-hf-to-gguf.py ./original_model/ --outtype f16 --outfile ./quantized_model/FP16.gguf 4. Quantizing the Model With the model in the correct format, we proceed to the quantization step. Here, we're using a quantization method called , which is specified in the list. This method quantizes the model to 4-bit precision with knowledge distillation and mapping techniques for better performance: q4_k_m methods import os

methods = ["q4_k_m"]
quantized_path = "./quantized_model/"

for m in methods:
    qtype = f"{quantized_path}/{m.upper()}.gguf"
    os.system("./llama.cpp/quantize "+quantized_path+"/FP16.gguf "+qtype+" "+m) 5. Testing the Quantized Model After quantization, it's important to test the model to ensure it performs as expected. The following command runs the quantized model with a sample prompt from a file, allowing you to assess its output quality: ! ./llama.cpp/main -m ./quantized_model/Q4_K_M.gguf -n 90 --repeat_penalty 1.0 --color -i -r "User: " -f llama.cpp/prompts/chat-with-bob.txt 6. Sharing Your Quantized Model Finally, if you wish to share your quantized model with the community, you can upload it to Hugging Face Hub using the and functions. This step involves creating a new repository for your quantized model and uploading the file: HfApi upload_file .gguf from huggingface_hub import HfApi, create_repo, upload_file

model_path = "./quantized_model/Q4_K_M.gguf"
repo_name = "gemma-2b-it-GGUF-quantized"
repo_url = create_repo(repo_name, private=False)

api = HfApi()
api.upload_file(
    path_or_fileobj=model_path,
    path_in_repo="Q4_K_M.gguf",
    repo_id="yourusername/gemma-2b-it-GGUF-quantized",
    repo_type="model",
) Make sure to replace with your actual Hugging Face username! "yourusername" Wrapping Up Congratulations! You've just learned how to quantize a large language model using llama.cpp. This process not only helps in deploying models to resource-constrained environments but also in reducing computational costs for inference. By sharing your quantized models, you contribute to a growing ecosystem of efficient AI models accessible to a broader audience. Quantization is a powerful tool in the machine learning practitioner's toolkit. Happy quantizing!

Quantizing Large Language Models With llama.cpp: A Clean Guide for 2024

How to Run Your Own Local LLM: Updated for 2024 - Version 2

After my latest post about how to build your own RAG and run it locally, today, we're taking it a step further by not only implementing the conversational abilities of large language models but also adding listening and speaking capabilities. The idea is straightforward: we are going to create a voice assistant reminiscent of Jarvis or Friday from the iconic Iron Man movies, which can operate offline on your computer. Since this is an introductory tutorial, I will implement it in Python and keep it simple enough for beginners. Lastly, I will provide some guidance on how to scale the application. Techstack First, you should set up a virtual Python environment. You have several options for this, including pyenv, virtualenv, poetry, and others that serve a similar purpose. Personally, I'll use Poetry for this tutorial due to my personal preferences. Here are several crucial libraries you'll need to install: : For a visually appealing console output. rich : A robust tool for speech-to-text conversion. openai-whisper : A cutting-edge library for text-to-speech synthesis, ensuring high-quality audio output. suno-bark : A straightforward library for interfacing with Large Language Models (LLMs). langchain , , and : Essential for audio recording and playback. sounddevice pyaudio speechrecognition For a detailed list of dependencies, refer to the link . here The most critical component here is the Large Language Model (LLM) backend, for which we will use Ollama. is widely recognized as a popular tool for running and serving LLMs offline. If Ollama is new to you, I recommend checking out my previous article on offline RAG: Basically, you just need to download the Ollama application, pull your preferred model, and run it. Ollama "Build Your Own RAG and Run It Locally: Langchain + Ollama + Streamlit." Architecture Okay, if everything has been set up, let's proceed to the next step. Below is the overall architecture of our application, which fundamentally comprises 3 main components: : Utilizing , we convert spoken language into text. Whisper's training on diverse datasets ensures its proficiency across various languages and dialects. Speech Recognition OpenAI's Whisper : For the conversational capabilities, we'll employ the Langchain interface for the model, which is served using Ollama. This setup promises a seamless and engaging conversational flow. Conversational Chain Llama-2 : The transformation of text to speech is achieved through , a state-of-the-art model from Suno AI, renowned for its lifelike speech production. Speech Synthesizer Bark The workflow is straightforward: record speech, transcribe to text, generate a response using an LLM, and vocalize the response using Bark. Sequence diagram for voice assistant with Whisper, Ollama, and Bark. Implementation The implementation begins with crafting a based on Bark, incorporating methods for synthesizing speech from text and handling longer text inputs seamlessly as follows: TextToSpeechService import nltk
import torch
import warnings
import numpy as np
from transformers import AutoProcessor, BarkModel

warnings.filterwarnings(
    "ignore",
    message="torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.",
)


class TextToSpeechService:
    def __init__(self, device: str = "cuda" if torch.cuda.is_available() else "cpu"):
        """
        Initializes the TextToSpeechService class.
        Args:
            device (str, optional): The device to be used for the model, either "cuda" if a GPU is available or "cpu".
            Defaults to "cuda" if available, otherwise "cpu".
        """
        self.device = device
        self.processor = AutoProcessor.from_pretrained("suno/bark-small")
        self.model = BarkModel.from_pretrained("suno/bark-small")
        self.model.to(self.device)

    def synthesize(self, text: str, voice_preset: str = "v2/en_speaker_1"):
        """
        Synthesizes audio from the given text using the specified voice preset.
        Args:
            text (str): The input text to be synthesized.
            voice_preset (str, optional): The voice preset to be used for the synthesis. Defaults to "v2/en_speaker_1".
        Returns:
            tuple: A tuple containing the sample rate and the generated audio array.
        """
        inputs = self.processor(text, voice_preset=voice_preset, return_tensors="pt")
        inputs = {k: v.to(self.device) for k, v in inputs.items()}

        with torch.no_grad():
            audio_array = self.model.generate(**inputs, pad_token_id=10000)

        audio_array = audio_array.cpu().numpy().squeeze()
        sample_rate = self.model.generation_config.sample_rate
        return sample_rate, audio_array

    def long_form_synthesize(self, text: str, voice_preset: str = "v2/en_speaker_1"):
        """
        Synthesizes audio from the given long-form text using the specified voice preset.
        Args:
            text (str): The input text to be synthesized.
            voice_preset (str, optional): The voice preset to be used for the synthesis. Defaults to "v2/en_speaker_1".
        Returns:
            tuple: A tuple containing the sample rate and the generated audio array.
        """
        pieces = []
        sentences = nltk.sent_tokenize(text)
        silence = np.zeros(int(0.25 * self.model.generation_config.sample_rate))

        for sent in sentences:
            sample_rate, audio_array = self.synthesize(sent, voice_preset)
            pieces += [audio_array, silence.copy()]

        return self.model.generation_config.sample_rate, np.concatenate(pieces) : The class takes an optional parameter, which specifies the device to be used for the model (either if a GPU is available, or ). It loads the Bark model and the corresponding processor from the pre-trained model. You can also use the large version by specifying for the model loader. Initialization ( __init__ ) device cuda cpu suno/bark-small suno/bark : This method takes a input and a parameter, which specifies the voice to be used for the synthesis. You can check out other value . It uses the to prepare the input text and the voice preset, and then generates the audio array using the method. The generated audio array is converted to a NumPy array and the sample rate is returned along with the audio array. Synthesize ( synthesize ) text voice_preset voice_preset here processor model.generate() : This method is used for synthesizing longer text inputs. It first tokenizes the input text into sentences using the function. For each sentence, it calls the method to generate the audio array. It then concatenates the generated audio arrays, with a short silence (0.25 seconds) added between each sentence. Long-form Synthesize ( long_form_synthesize ) nltk.sent_tokenize synthesize Now that we have the set up, we need to prepare the Ollama server for the large language model (LLM) serving. To do this, you'll need to follow these steps: TextToSpeechService : Run the following command to download the latest Llama-2 model from the Ollama repository: Pull the latest Llama-2 model ollama pull llama2 . : If the server is not yet started, execute the following command to start it: . Start the Ollama server ollama serve Once you've completed these steps, your application will be able to use the Ollama server and the Llama-2 model to generate responses to user input. Next, we'll move to the main application logic. First, we need to initialize the following components: : We'll use the Rich library to create a better interactive console for the user within the terminal. Rich Console : We'll initialize a Whisper speech recognition model which is a state-of-the-art open-source speech recognition system developed by OpenAI. We'll use the base English model ( ) for transcribing user input. Whisper Speech-to-Text base.en : We'll initialize a Bark text-to-speech synthesizer instance, which was implemented above. Bark Text-to-Speech : We'll use the built-in from the Langchain library which provides a template for managing the conversational flow. We'll configure it to use the Llama-2 language model with the Ollama backend. Conversational Chain ConversationalChain import time
import threading
import numpy as np
import whisper
import sounddevice as sd
from queue import Queue
from rich.console import Console
from langchain.memory import ConversationBufferMemory
from langchain.chains import ConversationChain
from langchain.prompts import PromptTemplate
from langchain_community.llms import Ollama
from tts import TextToSpeechService

console = Console()
stt = whisper.load_model("base.en")
tts = TextToSpeechService()

template = """
You are a helpful and friendly AI assistant. You are polite, respectful, and aim to provide concise responses of less 
than 20 words.
The conversation transcript is as follows:
{history}
And here is the user's follow-up: {input}
Your response:
"""
PROMPT = PromptTemplate(input_variables=["history", "input"], template=template)
chain = ConversationChain(
    prompt=PROMPT,
    verbose=False,
    memory=ConversationBufferMemory(ai_prefix="Assistant:"),
    llm=Ollama(),
) Now, let's define the necessary functions: : This function runs in a separate thread to capture audio data from the user's microphone using the . The callback function is called whenever new audio data is available, and it puts the data into a for further processing. record_audio sounddevice.RawInputStream data_queue : This function utilizes the Whisper instance to transcribe the audio data from the into text. transcribe data_queue : This function feeds the current conversation context to the Llama-2 language model (via the Langchain ) and retrieves the generated text response. get_llm_response ConversationalChain : This function takes the audio waveform generated by the Bark text-to-speech engine and plays it back to the user using a sound playback library (e.g., ). play_audio sounddevice def record_audio(stop_event, data_queue):
    """
    Captures audio data from the user's microphone and adds it to a queue for further processing.
    Args:
        stop_event (threading.Event): An event that, when set, signals the function to stop recording.
        data_queue (queue.Queue): A queue to which the recorded audio data will be added.
    Returns:
        None
    """
    def callback(indata, frames, time, status):
        if status:
            console.print(status)
        data_queue.put(bytes(indata))

    with sd.RawInputStream(
        samplerate=16000, dtype="int16", channels=1, callback=callback
    ):
        while not stop_event.is_set():
            time.sleep(0.1)

def transcribe(audio_np: np.ndarray) -> str:
    """
    Transcribes the given audio data using the Whisper speech recognition model.
    Args:
        audio_np (numpy.ndarray): The audio data to be transcribed.
    Returns:
        str: The transcribed text.
    """
    result = stt.transcribe(audio_np, fp16=False)  # Set fp16=True if using a GPU
    text = result["text"].strip()
    return text

def get_llm_response(text: str) -> str:
    """
    Generates a response to the given text using the Llama-2 language model.
    Args:
        text (str): The input text to be processed.
    Returns:
        str: The generated response.
    """
    response = chain.predict(input=text)
    if response.startswith("Assistant:"):
        response = response[len("Assistant:") :].strip()
    return response

def play_audio(sample_rate, audio_array):
    """
    Plays the given audio data using the sounddevice library.
    Args:
        sample_rate (int): The sample rate of the audio data.
        audio_array (numpy.ndarray): The audio data to be played.
    Returns:
        None
    """
    sd.play(audio_array, sample_rate)
    sd.wait() Then, we define the main application loop. The main application loop guides the user through the conversational interaction as follows: The user is prompted to press Enter to start recording their input. Once the user presses Enter, the function is called in a separate thread to capture the user's audio input. record_audio When the user presses Enter again to stop the recording, the audio data is transcribed using the function. transcribe The transcribed text is then passed to the function, which generates a response using the Llama-2 language model. get_llm_response The generated response is printed to the console and played back to the user using the function. play_audio if __name__ == "__main__":
    console.print("[cyan]Assistant started! Press Ctrl+C to exit.")

    try:
        while True:
            console.input(
                "Press Enter to start recording, then press Enter again to stop."
            )

            data_queue = Queue()  # type: ignore[var-annotated]
            stop_event = threading.Event()
            recording_thread = threading.Thread(
                target=record_audio,
                args=(stop_event, data_queue),
            )
            recording_thread.start()

            input()
            stop_event.set()
            recording_thread.join()

            audio_data = b"".join(list(data_queue.queue))
            audio_np = (
                np.frombuffer(audio_data, dtype=np.int16).astype(np.float32) / 32768.0
            )

            if audio_np.size > 0:
                with console.status("Transcribing...", spinner="earth"):
                    text = transcribe(audio_np)
                console.print(f"[yellow]You: {text}")

                with console.status("Generating response...", spinner="earth"):
                    response = get_llm_response(text)
                    sample_rate, audio_array = tts.long_form_synthesize(response)

                console.print(f"[cyan]Assistant: {response}")
                play_audio(sample_rate, audio_array)
            else:
                console.print(
                    "[red]No audio recorded. Please ensure your microphone is working."
                )

    except KeyboardInterrupt:
        console.print("\n[red]Exiting...")

    console.print("[blue]Session ended.") Result Once everything is put down together, we can run the application as shown in the video above. The application runs quite slowly on my MacBook because the Bark model is large, even in its smaller version. Therefore, I have slightly sped up the video. For those with a CUDA-enabled computer, it might run faster. Here are the key features of our application: : Users can start and stop recording their voice input, and the assistant responds by playing back the generated audio. Voice-based interaction The assistant maintains the context of the conversation, enabling more coherent and relevant responses. The use of the Llama-2 language model allows the assistant to provide concise and focused responses. Conversational context: For those aiming to elevate this application to a production-ready status, the following enhancements are recommended: : Incorporate optimized versions of the models, such as whisper.cpp, llama.cpp, and bark.cpp, which are designed to boost performance, especially on lower-end computers. Performance Optimization : Implement a system that allows users to customize the bot's persona and prompt, enabling the creation of different types of assistants (e.g., personal, professional, or domain-specific). Customizable Bot Prompts : Develop a user-friendly GUI to enhance the overall user experience, making the application more accessible and visually appealing. Graphical User Interface (GUI) : Expand the application to support multimodal interactions, such as the ability to generate and display images, diagrams, or other visual content in addition to the voice-based responses. Multimodal Capabilities Finally, we have completed our simple voice assistant application, full code can be found at: . This combination of speech recognition, language modeling, and text-to-speech technologies demonstrates how we can build something that sounds difficult but can actually run on your computer. Let's enjoy coding, and don't forget to subscribe to so you don't miss the latest in AI and programming articles. https://github.com/vndee/local-talking-llm my blog Also published here

How to Build Your Own Voice Assistant and Run it Locally Using Whisper + Ollama + Bark

কীভাবে আপনার নিজের ভয়েস সহকারী তৈরি করবেন এবং হুইস্পার + ওল্লামা + বার্ক ব্যবহার করে স্থানীয়ভাবে এটি চালাবেন

So erstellen Sie Ihren eigenen Sprachassistenten und führen ihn lokal mit Whisper + Ollama + Bark aus

Cómo crear su propio asistente de voz y ejecutarlo localmente usando Whisper + Ollama + Bark

Comment créer votre propre assistant vocal et l'exécuter localement à l'aide de Whisper + Ollama + Bark

अपना खुद का वॉयस असिस्टेंट कैसे बनाएं और इसे व्हिस्पर + ओलामा + बार्क का उपयोग करके स्थानीय रूप से कैसे चलाएं

独自の音声アシスタントを構築し、Whisper + Ollama + Bark を使用してローカルで実行する方法

나만의 음성 어시스턴트를 구축하고 Whisper + Ollama + Bark를 사용하여 로컬에서 실행하는 방법

Como construir seu próprio assistente de voz e executá-lo localmente usando Whisper + Ollama + Bark

Как создать собственный голосовой помощник и запустить его локально с помощью Whisper + Ollama + Bark

Kendi Sesli Asistanınızı Oluşturma ve Whisper + Ollama + Bark Kullanarak Yerel Olarak Çalıştırma

Cách xây dựng trợ lý giọng nói của riêng bạn và chạy nó cục bộ bằng Whisper + Ollama + Bark

如何构建您自己的语音助手并使用 Whisper + Ollama + Bark 在本地运行

Intro Running open-source AI models locally on our own computers gives us privacy, endless possibilities of tinkering, and freedom from large corporations. It is almost a matter of free speech. For us GPU-poor, however, having our own AI computer seems to be a pricey dream. ? $3200, ouch! ? $1850; that hurts even if you can get one. ? Starting at $1200, still too much. Macbook M3 Max Nvidia 4090 Microsoft Surface Laptop 6 What if I tell you that you can get a useful AI computer for $300? Interested? You do need to supply your own monitor, keyboard, and mouse. And you need a bit of tinkering around the Linux operating system, drivers, middleware, and configurations. To clarify, we are NOT talking about “training” or “fine-tuning” large generative AI models. We will focus on how to run open-source LLM (large language models such as 7B) locally, as well as generating images using . LLama 2 Stable Diffusion Now, let’s continue. What Makes a Good (and Cheap) AI Computer? Let’s assume one of the main use cases for a home AI computer is running or LLM inference. This task actually does not need a GPU at all since it can all be done in CPU. is an open-source software that enables very fast LLM inference using normal CPU. It was originally designed for the Macbook with an Apple M-series CPU, but it does work on Intel/AMD CPUs as well. large language models llama.cpp However, you do need the following for a faster inference speed. Otherwise, you will be like watching hair grow on your palm while the LLM spits out one token at a time. Fast CPU to memory bandwidth Faster DRAM (at least DDR4, DDR5 will be even better) A lot of memory (like 16GB minimal), especially if you want to run larger models (beyond 7B) For image generation with Stable Diffusion, you do need GPU power. However, you don’t have to have a very fancy GPU for that. You can leverage the integrated GPU already in your home computers: All Macs with M1/M2/M3 CPU, which integrates CPU, GPU, and high-speed memory (they are really good, but due to price are excluded from this particular article) AMD APU (e.g., Ryzen 7 5700U), which integrates CPU and GPU for budget-friendly mini-PCs. This will be the focus of this article. Intel CPU (e.g., Core i5-1135G7), which also integrates CPU and GPU. They are slightly above the $300 budget for the entire mini-PC, but readers are welcome to explore them further on their own. And the $300 AI Computer Is? An AMD-based Mini PC with the following specs usually sells for less than $300. I don’t want to endorse any particular brand, so you can search yourself: 5800H (8C/16T, up to 4.4GHz) AMD Ryzen 7 16GB RAM DDR4 (32GB recommended) 512GB NVME M.2 SSD I splurged a bit and opted for the $400 model with 32GB RAM and 1TB SSD (everything else equal). The main reason is that I do research on open-source LLMs and would like to run bigger models, in addition to running Stable Difusion. But you should be able to do almost everything in this article with the $300 computer. Prep 1: Allocate Enough iGPU Memory For AMD APUs like the , memory is shared between CPU and iGPU (integrated GPU). In my case, I have 32GB RAM total, but the default allocation for iGPU was only 3GB! This varies from computer to computer and is configured in BIOS during manufacturing. Ryzen 7 5800H You need to change that depending on your main use case: If you only need to run LLM inference, you can skip this entire prep step. Since LLM inference will only need to use CPU, and you should save most RAM for the CPU so you can run larger LLM models. If you need to run , especially (1024x1024), you need to allocate as much RAM for the iGPU as the system allows (typically half of total RAM) Stable Diffusion SDXL In my case, I want to run both Stable Diffusion XL and LLM inference on the same mini PC. Therefore, I would like to allocate 16GB (out of 32GB total) for the GPU. You can achieve this by changing the settings in BIOS. Typically, there is an upper limit, and the default setting might be much lower than the upper limit. On my computer, the upper limit was 16GB, or half of the total RAM available. Good BIOS If your computer’s BIOS supports such settings, go ahead and change to your desired number. My BIOS has no such setting. Poor BIOS: Use the Universal AMD tool If your BIOS does not have this setting, then please follow the nice instruction “Unlocking GPU Memory Allocation on AMD Ryzen™ APU?”  by Winston Ma. I tried it and it worked well, so now I have 16GB VRAM. https://winstonhyypia.medium.com/amd-apu-how-to-modify-the-dedicated-gpu-memory-e27b75905056 Prep 2: Install Drivers & Middleware Align the Stars AMD’s (Radeon Open Compute platform), comparable to Nvidia’s , is a suite of drivers and middleware to enable developers to utilize the power of ADM’s GPUs. And typically, AI applications need ROCm to get GPU acceleration. ROCm CUDA In order to install and make AMD’s ROCm work, you have to make sure that the versions of GPU hardware, Linux distro, kernel, python, HIP driver, ROCm library, and pytorch are compatible. If you want the least pain and maximum possibility of first-time success, stick with the recommended and verified combinations. Prerequisite Please check the following link to get the compatible Linux OS and kernel versions, and install them. Initially, I made the mistake of just installing my favorite Linux OS and default Linux kernel, and it was a big pain to walk backward to resolve compatibility issues. You can avoid this pain by just using the officially supported combinations. https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html ROCm Installation https://rocm.docs.amd.com/en/latest/deploy/linux/os-native/install.html If the entire installation finishes well, you can type in , and something like this will show (I only snipped the most relevant parts in highlighted yellow): rocminfo ROCk module is loaded ===================== HSA System Attributes ===================== Runtime Version:         1.1 System Timestamp Freq.:  1000.000000MHz Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) Machine Model:           LARGE System Endianness:       LITTLE Mwaitx:                  DISABLED DMAbuf Support:          YES ========== HSA Agents ========== ******* Agent 1 ******* Name:                    AMD Ryzen 7 5800H with Radeon Graphics Uuid:                    CPU-XX Marketing Name:          AMD Ryzen 7 5800H with Radeon Graphics Vendor Name:             CPU Pool Info: Pool 1 Segment:                 GLOBAL; FLAGS: COARSE GRAINED Size:                    16777216(0x1000000) KB Python Environment Python dependency can be quite tricky, so it is good practice to set up a proper environment. You can use either or for this purpose. conda venv source venv/bin/activate conda activate llm Pytorch https://pytorch.org/get-started/locally/ pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm5.7 HSA Overwrite The following is specific to APU’s with integrated graphics. Even though they are not officially supported by ROCm, the following proved to work. export HSA_OVERRIDE_GFX_VERSION=9.0.0 How to Verify Now, after all the complicated steps, let’s test if ROCm is working with Torch. And you can see that ROCm is “pretending” to be CUDA for the purpose of Pytorch. https://rocm.docs.amd.com/en/latest/how_to/pytorch_install/pytorch_install.html#test-the-pytorch-installation python3 -c 'import torch' 2> /dev/null && echo 'Success' || echo 'Failure' Success python3 -c 'import torch; print(torch.cuda.is_available())' True LLM Inference Let’s start with something easy for our newly configured $300 AI computer: running a large language model locally. We can choose one of the popular open-source modes: with 7B parameters that is optimized for chat. LLaMA 2 In addition, you can also try small LLMs from , , , and . More good quality LLMs can be found here on the very useful “chatbot arena leaderboard” by UC Berkeley’s . Mistral QWen Zephyr Vicuna LMSYS labs https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard Llama.cpp We will be using , which is initially optimized for CPU and later supports GPU as well. In my experience, LLM inference works well on the CPU, and there is little to gain with a modest GPU such as the ones integrated inside the $300 AI machine. llama.cpp https://github.com/ggerganov/llama.cpp First, you need to install and . And then follow the steps to compile and install llama.cpp. wget git sudo apt-get install build-essential git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp make Download Model Weights In order to run the LLMs on our inexpensive machine instead of cloud servers with expensive GPUs, we need to use a “compressed” version of the models so they can fit into the RAM space. For a simple example, a LLaMA-2 7B model has 7B parameters, each represented by float16 (2 bytes). Float 16: 14B bytes or 14GB which will not fit into our 8GB RAM. Quantized to 4-bit: 3.5B bytes or 3.5GB which can now fit into our 8GB RAM. Also, the file format should be . So. in our example, you need to download the weights in this file: gguf https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/blob/main/llama-2-7b-chat.Q4_0.gguf Test on AMD Mini PC First, we tested it on the AMD mini PC, and we achieved about 10 tokens per second. This is actually quite decent, and you can carry on a chat with the LLM without too much waiting. System config: AMD Ryzen 5800H 32GB RAM Command line instruction: ./main -m models/llama-2-7b-chat.Q4_0.gguf --color -ins -n 512 --mlock llama_print_timings:        load time =     661.10 ms llama_print_timings:      sample time =     234.73 ms /   500 runs   (    0.47 ms per token,  2130.14 tokens per second) llama_print_timings: prompt eval time =    1307.11 ms /    32 tokens (   40.85 ms per token,    24.48 tokens per second) llama_print_timings:        eval time =   50090.22 ms /   501 runs   (   99.98 ms per token,    10.00 tokens per second) llama_print_timings:       total time =   64114.27 ms Test on Intel Mini PC Next, we tested on an Intel mini PC, and we achieved about 1.5 tokens per second. This is a bit too slow for a fruitful chat session. It is not a fair comparison, since the Intel N5105 is clearly weaker than AMD 5800H. But that is the only Intel mini PC in my possession. If you use the more powerful Intel CPU (e.g., Core i5-1135G7) you should get comparable results. Please report your findings in the comments below. System config: 11th Gen 4 Cores N5105 (Up to 2.9Ghz) 4 Cores and 4 Threads 16GB RAM (2GB VRAM for iGPU) ./main -m models/llama-2-7b-chat.Q4_0.gguf -ins --color -n 512 --mlock llama_print_timings:        load time = 14490.05 ms llama_print_timings:      sample time =   171.53 ms /    97 runs   (    1.77 ms per token,   565.49 tokens per second) llama_print_timings: prompt eval time = 21234.29 ms /    33 tokens (  643.46 ms per token,     1.55 tokens per second) llama_print_timings:        eval time = 75754.03 ms /    98 runs   (  773.00 ms per token,     1.29 tokens per second) Stable Diffusion Installation https://github.com/AUTOMATIC1111/stable-diffusion-webui And pay attention to this page as well, in regards to AMD ROCm https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Install-and-Run-on-AMD-GPUs Quick Start export HSA_OVERRIDE_GFX_VERSION=9.0.0 source venv/bin/activate ./webui.sh --upcast-sampling --skip-torch-cuda-test --precision full --no-half Stable Diffusion 1.5 Test ./webui.sh --upcast-sampling --skip-torch-cuda-test --precision full --no-half Test 1 Prompt: “horse in forest” Steps: 20, Sampler: DDIM, CFG scale: 7, Seed: 519288240, Size: 512x512, Model hash: 6ce0161689, Model: v1-5-pruned-emaonly, Version: v1.6.0 Time taken: 1 min. 8.3 sec. Stable Diffusion XL 1.0 Test SDXL (max resolution 1024x1024) recommends at least 12GB VRAM, so you definitely need to get the Prep 1 step done to allocate 16GB VRAM for iGPU. So, this task is only possible with the $400 mini PC. ./webui.sh --upcast-sampling Test 1: Prompt: “horse in forest” Steps: 20, Sampler: DDIM, CFG scale: 7, Seed: 1102941451, Size: 1024x768, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, Version: v1.6.0 Time taken: 7 min. 41 sec Test 2: Prompt: “young taylor swift in red hoodie riding a horse in forest” Negative prompt: deformities, deformity, deformed eyes, deformed teeth, deformed fingers, deformed face, deformed hands, deformed Steps: 20, Sampler: DDIM, CFG scale: 7, Seed: 2960206663, Size: 1024x1024, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, Version: v1.6.0 Time taken: 6 min. 12.3 sec. Windows 11 and AMD/directml Although this article focuses on Linux operating systems, you can get Stable Diffusion working in Windows too. Here are my experiments: https://github.com/lshqqytiger/stable-diffusion-webui-directml First, you need to install Python 3.10.6. Add Python 3.10.6 director to PATH. Important: The Python path has to be the top path https://realpython.com/add-python-to-path/ Install git and git clone repo Run webui-user.bat from the file explorer Test 1: Prompt: “horse in forest” Settings: DPM++ 2M Karras, 512x512, sampling steps 20 Time taken: 1m19s Conclusions So, are you having fun running your own generative AI models on your new $300 mini PC? I hope you do. Open-source AI models running on personal devices is one of the most exciting areas for tinkers since none of us will have the massive GPU pool to actually train a foundational model. This will enable a new generation of apps that are both super smart while still preserving our data privacy. What next? Run it on even smaller embedded devices: e.g., Raspberry Pi Run it on your smartphones (llama.cpp does support iOS and Android) And happy tinkering with AI, open source, and on-device!

How to Build a $300 AI Computer for the GPU-Poor

GPU-দরিদ্রদের জন্য $300 এআই কম্পিউটার কীভাবে তৈরি করবেন

So bauen Sie einen 300-Dollar-KI-Computer für GPU-arme Menschen

Cómo construir una computadora con inteligencia artificial de $ 300 para los pobres en GPU

Comment construire un ordinateur IA à 300 $ pour les pauvres en GPU

जीपीयू-खराब के लिए $300 का एआई कंप्यूटर कैसे बनाएं

GPU が苦手な人向けに 300 ドルの AI コンピューターを構築する方法

GPU가 부족한 사람들을 위해 300달러짜리 AI 컴퓨터를 만드는 방법

Como construir um computador com IA de US$ 300 para os pobres em GPU

Как построить компьютер с искусственным интеллектом за 300 долларов для тех, у кого мало графических процессоров

GPU'su Yoksullar için 300 Dolarlık Yapay Zekalı Bilgisayar Nasıl Oluşturulur

Cách xây dựng một máy tính AI trị giá 300 USD cho người có GPU kém

如何为 GPU 匮乏的人打造一台 300 美元的人工智能计算机

How to Run Your Own Local LLM: Updated for 2024 - Version 1

কিভাবে আপনার নিজের স্থানীয় LLM চালাবেন (2024 এর জন্য আপডেট করা হয়েছে)

So führen Sie Ihren eigenen lokalen LLM durch (Aktualisiert für 2024)

Cómo ejecutar su propio LLM local (actualizado para 2024)

Comment gérer votre propre LLM local (mis à jour pour 2024)

अपना खुद का स्थानीय एलएलएम कैसे चलाएं (2024 के लिए अपडेट किया गया)

独自のローカル LLM を運営する方法 (2024 年更新)

자신만의 지역 LLM을 운영하는 방법(2024년 업데이트)

Como executar seu próprio LLM local (atualizado para 2024)

Как запустить собственный местный LLM (обновлено для 2024 г.)

Kendi Yerel LLM'nizi Nasıl Yönetirsiniz (2024 için Güncellendi)

Cách chạy LLM địa phương của riêng bạn (Cập nhật năm 2024)

如何开办自己的本地法学硕士 (2024 年更新)

Introduction If you haven’t read my previous article then you may find it beneficial to review, as I’ve defined terms and explained the means by which I came to various practices and beliefs. PrivateGPT for Book Summarization: Testing and Ranking Configuration Variables If you did read that article, then you will be aware that I’ve been refining my processes, for a few months, using Large Language Models (LLM) for the purpose of summarizing books. I measured a series of parameters including prompt templates, system prompts, user prompts, etcetera. From that preliminary round of model rankings and collecting data on the use of configuration variables, (Tho OpenChat 3.5 0106 is a strong contender) I found to produce the highest quality bulleted notes, and have been searching for one to best it, that fits on my 12GB 3060, ever since. mistral-7b-instruct-v0.2.Q8_0.gguf For this ranking, I’m using that base of knowledge to assess a variety of leading 7b models. This time I’m using , as I find it simpler to use and quite performant. Ollama I chose the following models because I found them ranking above on various leader-boards, or were self-proclaimed as best 7b. Mistral 7b Instruct 0.2 (chat templates tested in parenthesis) openchat-3.5-0106.Q8_0.gguf (OpenChat) snorkel-mistral-pairrm-dpo.Q8_0.gguf (Mistral) dolphin-2.6-mistral-7b.Q8_0.gguf (Mistral) supermario-v2.Q8_0.gguf (ChatML) openhermes-2.5-mistral-7b.Q8_0.gguf (ChatML) openhermes-2.5-neural-chat-7b-v3-1-7b.Q8_0.gguf (ChatML) openhermes-2.5-neural-chat-v3-3-slerp.Q8_0.gguf (ChatML) WestLake-7B-v2-Q8_0.gguf (ChatML, Mistral) MBX-7B-v3-DPO.q8_0.gguf (ChatML, Mistral) neuralbeagle14-7b.q8_0.gguf (ChatML, Mistral) omnibeagle-7b-q8_0.gguf (ChatML, Mistral) For some models, where I wasn’t getting the desired results, as they are mostly Mistral derived, I tested the Mistral template even though they list ChatML as their preferred input. Bullet Point Notes With Headings and Terms in Bold Write comprehensive bulleted notes summarizing the following text, with headings, terms, and key concepts in bold.\n\nTEXT: While GPT3.5 isn’t my personal baseline, it is something of an industry standard, and I would expect it to produce better results than most 7b Q8 GGUFs. While there are no key concepts of terms in bold, the headings in bold, and overall, this is quite easy to read compared to blocks of paragraphs. Also, whether or not we find terms in bold may depend on the input text itself, where a bullet point summary should include bolded headings. are always I’m Looking for Models That Produces Notes: faster with more detail, less filler with comparable detail with longer context (currently stretching these capabilities around 2.5k context) I see this as a fundamental task for any Instruct model. Ideally, developers will train their models to generate these types of ideal bulleted notes. I have tons of data, with some books trained already, but it’s relatively simple to generate these notes for a book (Using with the text semantically chunked, by hand, into parts below 2.5k tokens, each). Mistral 7b Instruct 0.2 If it’s a 300-600 page book, then it can usually be done in a single day, including pre and post-processing. Eventually, I might experiment with some fine-tuning in an attempt to improve their capacities myself. Ollama is a command-line tool for running GGUF, written in GO and depending upon . That’s right, Its fast! Its got lots of and supports OpenAI API format, so you can use it with your favorite apps, front-ends, and frameworks. Ollama Llama.cpp great integrations Ollama is really easy to use from the command-line and provides a simple interface for adding new models having their own particular parameters. Ollama Modelfiles include GGUF location, template, and parameters to a Model file, which it uses to save a copy of your chosen model, using your specified configuration. This makes it easy to demo various models without too much fussing around with parameters. Modelfiles I’ve kept the parameters the same for all models except the chat template, but I will share with you the template I’m using for each, so you can see precisely how I use the template. You can let me know if I’d get better results from the following models using a differently configured. Once you’ve made a modelfile (as shown below for each model tested) with your chosen LLM and parameters, getting is easy: ollama create mistralq8 -f Modelfile
ollama run mistralq8 "How is the weather in San Francisco?" I’ve left a more detailed on GitHub. walkthrough The Rankings Previously, I tried to give each ranking a score. It’s really hard to give a numerical score. In the future, I think I’ll try to get an LLM to rank the summaries. This time, I’ll just leave a comment on where it falls short, and what I like, without giving a numerical score to each model. I tested each of the following models on a single book chapter, divided into 6 chunks from 1900-3000 tokens each. I’ll share a representative example output from each, and the full data will be available on GitHub, as always. Mistral 7b Instruct 0.2 Q8 GGUF I’m sure you realize by now that, in my opinion, has the 7b to beat. Mistral Modelfile TEMPLATE """
<s></s>[INST] {{ .Prompt }} [/INST]
"""
PARAMETER num_ctx 8000
PARAMETER num_gpu -1
PARAMETER num_predict 4000 Mistral 7b Instruct v0.2 Result I won’t say that Mistral does it perfectly every single time, but more often than not, this is my result. And if you look back to the GPT3.5 response, you might agree that this is better. OpenChat 3.5 0106 Q8 GGUF I was pleasantly surprised by . Here is a model that claims to have the best 7b model, and at least is competitive with Mistral 7b. OpenChat’s 0106 Modelfile TEMPLATE """
GPT4 Correct User:  {{ .Prompt }}<|end_of_turn|>GPT4 Correct Assistant:
"""
PARAMETER num_ctx 8000
PARAMETER num_gpu -1
PARAMETER num_predict 4000 OpenChat 3.5 0106 Result In this small sample, it gave bold headings 4/6 times. Later, I will review it along with any other top contenders using a more detailed analysis. Snorkel Mistral Pairrm DPO Q8 GGUF Obviously, I’m biased, here, as Snorkel was trained on Mistral 7b Instruct 0.2. Regardless, I am cautiously optimistic and look forward to more releases from . Snorkel.ai Modelfile TEMPLATE """
<s></s>[INST] {{ .Prompt }} [/INST]
"""
PARAMETER num_ctx 8000
PARAMETER num_gpu -1
PARAMETER num_predict 4000 Snorkel Mistral Pairrm DPO Result 4/6 of these summaries are spot on, but others contain irregularities such as super long lists of key terms and headings instead of just bolding them inline as part of the summary. Dolphin 2.6 Mistral 7B Q8 GGUF Here is that’s well regarded. another mistral derivative Modelfile TEMPLATE """
<|im_start|>system
You are a helpful AI writing assistant.<|im_end|>
<|im_start|>user
{{ .Prompt }} <|im_end|>
<|im_start|>assistant
{{ .Response }}<|im_end|>
"""
PARAMETER num_ctx 8000
PARAMETER num_gpu -1
PARAMETER num_predict 4000 Dolphin 2.6 Mistral 7B Result This is another decent model that’s as good as Mistral 7b Instruct 0.2. Three out of 6 summaries gave proper format and bold headings, another had good format with no bold, but 2/6 were bad form all around. almost OpenHermes 2.5 Mistral-7B Q8 GGUF is quite popular, both on leaderboards and among “the people” in unassociated discord chats. I want it to be a leader in this ranking, but it’s just not. This model Modelfile TEMPLATE """
<|im_start|>system
You are a helpful AI writing assistant.<|im_end|>
<|im_start|>user
{{ .Prompt }} <|im_end|>
<|im_start|>assistant
{{ .Response }}<|im_end|>
"""
PARAMETER num_ctx 8000
PARAMETER num_gpu -1
PARAMETER num_predict 4000 OpenHermes 2.5 Mistral Result 3/6 results produce proper structure, but no bold text. One of them has both structure and bold text. The other two had more big blocks of text \ and poor structure. OpenHermes 2.5 Neural Chat 7b v3.1 7B Q8 GGUF I also tried a few high-ranking of OpenHermes 2.5 Mistral to see if I could get better results. Unfortunately, that was not the case. derivatives Modelfile TEMPLATE """
<|im_start|>system
You are a helpful AI writing assistant.<|im_end|>
<|im_start|>user
{{ .Prompt }} <|im_end|>
<|im_start|>assistant
{{ .Response }}<|im_end|>
"""
PARAMETER num_ctx 8000
PARAMETER num_gpu -1
PARAMETER num_predict 4000 OpenHermes 2.5 Neural Chat 7b v3.1 Result None of these results were desirable. OpenHermes 2.5 Neural-Chat v3.3 Slerp Q8 GGUF Whatever they did, these did not improve upon the original. derivatives Modelfile TEMPLATE """
<|im_start|>system
You are a helpful AI writing assistant.<|im_end|>
<|im_start|>user
{{ .Prompt }} <|im_end|>
<|im_start|>assistant
{{ .Response }}<|im_end|>
"""
PARAMETER num_ctx 8000
PARAMETER num_gpu -1
PARAMETER num_predict 4000 OpenHermes 2.5 neural-chat v3.3 Slerp Result It’s just getting worse with each new version! Super Mario V2 Q8 I wasn’t expecting much from Mario, but it shows some promise. Meanwhile, V3 and V4 are available, but I haven’t found GGUF for those, yet. Modelfile TEMPLATE """
<|im_start|>system
You are a helpful AI writing assistant.<|im_end|>
<|im_start|>user
{{ .Prompt }} <|im_end|>
<|im_start|>assistant
{{ .Response }}<|im_end|>
"""
PARAMETER num_ctx 8000
PARAMETER num_gpu -1
PARAMETER num_predict 4000 Super Mario V2 Result Its first result was deceptively good. However, each of the following summaries deviated from the desired pattern. I’ll be on the lookout for GGUF of the newer releases. You can see here we got blocks of paragraphs with an initial bolded heading. Not really what I asked for. Honorable Mentions (ChatML) - This one is actually producing a decent format but no bolded text. omnibeagle-7b (ChatML, Mistral) - Works better with the mistral template. neuralbeagle14-7b (ChatML) - I’ve seen worse WestLake-7B-v2 (ChatML) - No consistency in format. MBX-7B-v3-DPO Conclusion I wish I had better news to share. My ideal headline is that Unfortunately, that is not the case. there is an abundance of leading models that produce quality output when creating and it’s just so hard for me to choose among them. comprehensive bulleted note summaries, Maybe they outperform Mistral 0.2 in full form but only are trailing in GGUF format? I think it’s quite likely that none of our existing evals target this type of output, but I would certainly argue that it’s a task that any leading 7b gguf model should be able to manage. Another thing to consider is that Mistral 7b Instruct v0.2 came out soon after Mixtral, amidst a bunch of fanfare. I think that release slipped under the radar. In fact, many of the “leading” models I’ve looked at are based on 0.1 Mistral. Maybe things will change, and the world will realize that their best models still can’t top Mistral? Then again, maybe all those models are really good at all the other tasks I’m not asking for. Help Me Help You I have data, I have a pipeline, and I have an endless need to create bulleted note summaries. If you want to work with me, please reach out. You are also welcome to check out my GitHub, and try out your own version of this experiment. I’m happy to be proven wrong. check the data, Future Direction I would like to do a deeper analysis on the best models of this round, which are really the survivors of numerous previous rounds of rankings. Included in that analysis might be further experimentation with prompt templates, user prompt, system prompt. I’m also interested in trying these same models with an example summary included in its context.

Ranking 7b Q8 GGUF for Comprehensive Bulleted Notes With Ollama

ওল্লামার সাথে ব্যাপক বুলেটেড নোটের জন্য 7b GGUF র‌্যাঙ্কিং: হোম মডেল র‌্যাঙ্কিংয়ে যান, আপনি মাতাল!

Rang 7b GGUF für umfassende Aufzählungsnotizen mit Ollama: Go Home Model-Rangliste, du bist betrunken!

Clasificación 7b GGUF para notas completas con viñetas con Ollama: Clasificación de modelos Go Home, ¡estás borracho!

Classement 7b GGUF pour des notes à puces complètes avec Ollama : Go Home Model Rankings, You're Drunk !

ओलामा के साथ व्यापक बुलेटेड नोट्स के लिए रैंकिंग 7बी जीजीयूएफ: गो होम मॉडल रैंकिंग, आप नशे में हैं!

オラマとの包括的な箇条書きノートのランキング 7b GGUF: モデル ランキング、酔っぱらって帰れ!

Ollama: Go Home 모델 순위로 종합 글머리 기호 메모를 위한 GGUF 순위 7b, 당신은 취했어요!

Classificação 7b GGUF para notas abrangentes com marcadores com Ollama: Go Home Model Rankings, você está bêbado!

Рейтинг 7b GGUF за подробные маркированные заметки с Олламой: рейтинг моделей, идите домой, вы пьяны!

Ollama ile Kapsamlı Madde İşaretli Notlar için GGUF Sıralaması 7b: Eve Git Model Sıralaması, Sarhoşsun!

Xếp hạng 7b GGUF về ghi chú có dấu đầu dòng toàn diện với Ollama: Xếp hạng mô hình về nhà, bạn say rồi!

Ollama 综合项目符号注释排名 7b GGUF：回家模型排名，你醉了！

The last year was all about the race of commercial LLMs. Though the LLaMa leakage and further Meta actions on the open-source front were significant, and Hugging Face kept blooming as a platform for open source, this year starts with a powerful player stepping in with a real open-source approach. We speak, of course, about : by the Allen Institute for Artificial Intelligence. Newsletters such as AlphaSignal, TheSequence, Data Machina, Smol Talk, and Interconnects (Nathan Lambert is one of the authors of the OLMo paper) explained pretty well what’s the difference between almost open-source and truly open-source models. OLMo Accelerating the Science of Language Model, released The gist of it is that truly open source means that not only the weights of the model and inference code are released but truly the whole package: the training data, training and evaluation code, and a comprehensive framework for studying language modeling. Who is behind OLMo? What I found interesting is who stands behind the release. While EleutherAI’s Pythia and Big Science’s BLOOM previously set a precedent for releasing fully open-source models, the distinction with OLMo is its release by a true nonprofit organization — the Allen Institute for AI (AI2). AI2 was founded in 2014 by philanthropist and Microsoft co-founder Paul G. Allen, who committed to conducting high-impact research and engineering in artificial intelligence. He was also very interested in teaching machines “common sense.” And he funded this cause well. Once, I had a conversation with one of the top executives of AI2; the person said that thanks to Paul Allen’s financing structure, AI2 is well-funded, has no influence from large companies, and has no pressure to make money. AI2 is famous for conducting cutting-edge research in AI and aiming to influence the broader AI research community by releasing open-source software, datasets, and research findings. Projects like the academic search engine democratize access to information and accelerate scientific breakthroughs. Semantic Scholar Why OLMo is special The OLMo framework includes multiple training checkpoints, logs, exact datasets used, and a permissive license, establishing a new standard for openness in the field. They also don’t mind this model being used for commercial purposes. Unlike others, the researchers readily embrace openness, believing it outweighs the low misuse risk, as their models, not designed as chatbots, contribute to science rather than commercial products. Furthermore, they released ‘ ’ According to , ‘the name of the pretraining corpus, “Dolma,” stands for Data to feed OLMo’s Appetite.’ Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. Luca Soldani What also surprised me was that the authors highlighted the environmental impact of training large LMs, providing estimates of power consumption and carbon emissions. They advocate for transparency in reporting these impacts and emphasize the potential for open models like OLMo to mitigate future emissions by minimizing redundant model training. Great start to the year of open-source! News from The Usual Suspects © Google In their blog post, Google MobileDiffusion, a new approach for sub-second text-to-image generation on mobile devices, addressing the efficiency challenges of current large models. This model combines a pre-trained diffusion model with a GAN to enable one-step sampling during inference, resulting in a compact model size of 520M parameters capable of generating high-quality images rapidly on iOS and Android devices. The research includes an in-depth exploration of model architecture, optimizing for mobile deployment while maintaining image quality. announced Meta In , Mark Zuckerberg explained how Meta benefits from open source: Meta’s Q4 2023 earnings call yesterday “There are several strategic benefits. First, open source software is typically safer and more secure, as well as more compute efficient to operate due to all the ongoing feedback, scrutiny, and development from the community. This is a big deal because safety is one of the most important issues in AI. Efficiency improvements and lowering the compute costs also benefit everyone including us. Second, open source software often becomes an industry standard, and when companies standardize on building with our stack, that then becomes easier to integrate new innovations into our products. That’s subtle, but the ability to learn and improve quickly is a huge advantage and being an industry standard enables that. Third, open source is hugely popular with developers and researchers. We know that people want to work on open systems that will be widely adopted, so this helps us recruit the best people at Meta, which is a very big deal for leading in any new technology area. And again, we typically have unique data and build unique product integrations anyway, so providing infrastructure like Llama as open source doesn’t reduce our main advantages. This is why our long-standing strategy has been to open source general infrastructure and why I expect it to continue to be the right approach for us going forward.” — Mark Zuckerberg The freshest research papers, categorized for your convenience Language Modeling and Efficiency : Examines LLMs in mathematical reasoning, highlighting capabilities and challenges. Large Language Models for Mathematical Reasoning: Progresses and Challenges read the paper : Introduces an ∞-gram model for extensive text analysis. Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens read the paper : Proposes a sustainable and efficient LLM pre-training method. Scavenging Hyena: Distilling Transformers into Long Convolution Models read the paper : Presents a strategy for efficient LLM pre-training using rephrased web documents. Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling read the paper Advanced Reasoning and Contextual Understanding : Enhances multi-step reasoning in LLMs for improved accuracy and efficiency. Efficient Tool Use with Chain-of-Abstraction Reasoning read the paper : Improves performance on complex reasoning tasks through hierarchical text summarization. RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval read the paper : Investigates LLMs’ contextual understanding and the impact of compression techniques. Can Large Language Models Understand Context? read the paper Enhancements in AI Frameworks and Methodologies : Merges generative models with solvers for advanced concept learning. SymbolicAI: A framework for logic-based approaches combining generative models and solvers read the paper : Demonstrates the benefits of efficient exploration in LLM training. Efficient Exploration for LLMs read the paper : Discusses methods for improving LLM alignment with human preferences. Transforming and Combining Rewards for Aligning Large Language Models read the paper : Introduces a MoE approach for efficient multi-modal learning in LVLMs. MoE-LLaVA: Mixture of Experts for Large Vision-Language Models read the paper Novel Applications and Security Insights : Develops an autonomous agent for mobile devices using visual perception. Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception read the paper : Explores text diffusion as a competitive method for LLMs. Transfer Learning for Text Diffusion Models read the paper : Identifies a security issue in LLMs and proposes initial defense strategies. Weak-to-Strong Jailbreaking on Large Language Models read the paper In other newsletters A very interesting read about a new trend: VCs backing companies where the majority of revenues come from government contracts by Newcomer The research report gives interesting details about the types of security assessments Chinese AI labs are implementing to comply with generative AI regulations by ChinAI celebrates a large milestone towards democratizing AI inference with their Vulkan Kompute project that was adopted as one of the backends for the LLama.cpp and GPT4ALL frameworks. ML Engineer Newsletter Also published . here

FOD 39: Truly Open – We Explore Who Stands Behind OLMo's Release

If you just want to see the code - see . github What are we going to build and why? LLMs are a new hot piece of technology that everyone is experimenting with. Managed services like OpenAI are the cheapest and most convenient way to use them. You can use their LLMs only for a small fee and for approval to train new versions of your data (in some cases). In certain applications, it is required to run an on our own. You may want to process sensitive data (medical records or legal documents) or get great quality output in a language different than English. Sometimes, you have a specialized task that doesn’t require expensive big models from OpenAI. LLM Open-source LLMs are matching the quality of big players like OpenAI but still require a lot of expensive computing resources. There are smaller models available that can run on weaker hardware and perform well enough. As an experiment, let’s deploy a smaller open-source LLM on AWS lambda. Our goal is to learn more about LLMs and docker-based lambdas. We will also evaluate performance and cost to determine if any real-world applications are feasible. For this project, we are going to use the model, a 2.7 billion parameter LLM that matches the quality of outputs from 13B parameters or more open-source models. It was trained on a large dataset and is a viable model for many applications. From my experience, it hallucinates a lot but otherwise provides useful outputs. Its size is perfect for the AWS Lambda environment. Microsoft Phi-2 We will download the model from Huggingface and run it via package (bindings to the popular llama.cpp, heavily optimized model CPU runtime). We will use a smaller, quantized version, but even the full one should fit in Lambda memory. llama-cpp-python We will create an HTTP REST endpoint via the lambda URL mechanism for a given call: PROMPT="Create five questions for a job interview for a senior python software engineer position."

curl $LAMBDA_URL -d "{ \"prompt\": \"$PROMPT\" }" \
    | jq -r '.choices[0].text, .usage' (jq command used to decode returned JSON) Provides LLM output alongside execution details: Instruct: Create five questions for a job interview for a senior python software engineer position.
Output: Questions: 

1. What experience do you have in developing web applications?
2. What is your familiarity with different Python programming languages?
3. How would you approach debugging a complex Python program?
4. Can you explain how object-oriented programming principles can be applied to software development? 
5. In a recent project, you were responsible for managing the codebase of a team of developers. Can you discuss your experience with this process?

{
  "prompt_tokens": 21,
  "completion_tokens": 95,
  "total_tokens": 116
} For this tutorial, you need to know the basics of programming, docker, AWS, and Python. I am using an M-series Mac, so as a default, we’re going to aim at deployment to ARM AWS Lambda, as it simply costs less for more performance. Instructions on how to deploy the x86 version are included. Environment setup (AWS, Docker and Python) I won’t dive into the basics here. If you don’t have the tools installed, please follow these resources. You need an AWS account and AWS CLI installed and configured. Good tutorial from Google search: . tutorial Official docs, including if you’re using SSO: AWS official tutorial. You also need docker: docker docs. As an IDE, I am using Visual Studio Code. Set up lambda function with docker locally. Let’s start with getting the local environment running. If you need more information than I included here, check out the official AWS documentation. We will store every file in a single project directory without subfolders. First, we will need a basic Python lambda function handler. In your project folder, create a file called . lambda_function.py import sys

def handler(event, context):
    return "Hello from AWS Lambda using Python" + sys.version + "!" We will also create a file, in which we specify our dependencies. Let’s start with the AWS library for interacting with their services, just as an example. requirements.txt boto3 Then, we need to specify our docker image composition in file. Comments document what each line does. Dockerfile FROM public.ecr.aws/lambda/python:3.12

# Copy requirements.txt
COPY requirements.txt ${LAMBDA_TASK_ROOT}

# Install the specified packages
RUN pip install -r requirements.txt

# Copy function code
COPY lambda_function.py ${LAMBDA_TASK_ROOT}

# Set the CMD to your handler
CMD [ "lambda_function.handler" ] Finally, we will create a file to make our life easier when running and building the container. docker-compose.yml version: '3'

services:
  llm-lambda:
		image: llm-lambda
    build: .
    ports:
      - 9000:8080 OK, done! Make sure your docker engine is running, and type in the terminal: docker-compose up The container should be built and started. Our lambda is available under a long URL below (quirks of the official Amazon image). To test if the lambda is running, open another terminal window/tab and type in: curl "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{}' Now you can come back to the tab you ran docker-compose in and type ctrl-c to stop the container. You can put this test code in a bash script, as seen in the project repo. Also, it will be useful to create a .gitignore file if you’re going to version control this project. Run an LLM inside a container. Now that we have a working lambda let’s add some AI magic. To run an LLM, we need to add to our . llama-cpp-python requirements.txt boto3
llama-cpp-python To build it, we need to introduce a . This is because the default Amazon docker image doesn’t include the build tools required for llama-cpp. We are doing a pip install with flags in order to use multi-threaded optimizations. docker build stage CMAKE_ARGS="-DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=OpenBLAS" The code below also includes downloading the model. Community hero, the Bloke, is sharing compressed (quantized, less computationally intensive but lower quality) versions of models, ready to download . To make use of them, we can install the huggingface CLI and run the appropriate commands. You can switch the repository ( ) and the model ( ) to whatever you like if you want to deploy a different model. here TheBloke/phi-2-GGUF phi-2.Q4_K_M.gguf RUN pip install huggingface-hub && \
    mkdir model && \
    huggingface-cli download TheBloke/phi-2-GGUF phi-2.Q4_K_M.gguf --local-dir ./model --local-dir-use-symlinks False # Stage 1: Build environment using a Python base image
FROM python:3.12 as builder

# Install build tools
RUN apt-get update && apt-get install -y gcc g++ cmake zip

# Copy requirements.txt and install packages with appropriate CMAKE_ARGS
COPY requirements.txt .
RUN CMAKE_ARGS="-DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=OpenBLAS" pip install --upgrade pip && pip install -r requirements.txt

# Stage 2: Final image using AWS Lambda Python image
FROM public.ecr.aws/lambda/python:3.12

# Install huggingface-cli and download the model
RUN pip install huggingface-hub && \
    mkdir model && \
    huggingface-cli download TheBloke/phi-2-GGUF phi-2.Q4_K_M.gguf --local-dir ./model --local-dir-use-symlinks False

# Copy installed packages from builder stage
COPY --from=builder /usr/local/lib/python3.12/site-packages/ /var/lang/lib/python3.12/site-packages/

# Copy lambda function code
COPY lambda_function.py ${LAMBDA_TASK_ROOT}

CMD [ "lambda_function.handler" ] Let’s modify our lambda code to run LLM inference. Some more work is required to get the prompt out of the request body in the production environment. import base64
import json
from llama_cpp import Llama

# Load the LLM, outside the handler so it persists between runs
llm = Llama(
    model_path="./model/phi-2.Q4_K_M.gguf", # change if different model
    n_ctx=2048, # context length
    n_threads=6,  # maximum in AWS Lambda
)

def handler(event, context):
    print("Event is:", event)
    print("Context is:", context)

		# Locally the body is not encoded, via lambda URL it is
    try:
        if event.get('isBase64Encoded', False):
            body = base64.b64decode(event['body']).decode('utf-8')
        else:
            body = event['body']

        body_json = json.loads(body)
        prompt = body_json["prompt"]
    except (KeyError, json.JSONDecodeError) as e:
        return {"statusCode": 400, "body": f"Error processing request: {str(e)}"}

    output = llm(
        f"Instruct: {prompt}\nOutput:",
        max_tokens=512, 
        echo=True,
    )

    return {
        "statusCode": 200,
        "body": json.dumps(output)
    } OK, time to test! My docker setup has 8GB of RAM assigned, but it should run well under 4GB. Let’s rebuild the container and start it again. It will take more time as the model needs to download (~1.8GB). docker-compose up --build This time, let’s test with a real prompt. The first run will take more time as the model needs to load. curl "http://localhost:9000/2015-03-31/functions/function/invocations" \
    -d '{ "body": "{ \"prompt\": \"Generate a good name for a bakery.\" }" }' Example output: Instruct: Generate a good name for a bakery.
Output: Sugar Rush Bakery.

{
  "prompt_tokens": 15,
  "completion_tokens": 6,
  "total_tokens": 21
} Deploy to AWS Lambda With everything working locally, let’s finally deploy our LLM to AWS! To execute the deployment successfully, the following steps are required: : Determine necessary information such as AWS region, ECR repository name, Docker platform, and IAM policy file, and disable AWS CLI pager. Initial Setup : Optionally confirm that the AWS CLI is correctly configured. Verify AWS Configuration : ECR (Elastic Container Registry) Repository Management Determine if the specified ECR repository already exists. If absent, create a new ECR repository. : IAM Role Handling Check for the existence of a specific IAM role for Lambda. If not found, establish the IAM role and apply the policy. AWSLambdaBasicExecutionRole : Securely log Docker into the ECR registry using the retrieved login credentials. Docker-ECR Authentication : Utilize Docker Compose to build the Docker image, specifying the desired platform. Docker Image Construction : Label the Docker image appropriately for ECR upload. ECR Image Tagging : Transfer the tagged Docker image to the ECR. ECR Image Upload : Fetch the ARN linked to the specified IAM role. Acquire IAM Role ARN : Assess whether the Lambda function exists. Lambda Function Verification : Set parameters like timeout, memory allocation, and image URI for Lambda. Lambda Function Configuration : Lambda Function Deployment/Update For a new Lambda function, create it with defined settings and establish a public Function URL. For an existing Lambda function, update its code using the new Docker image URI. : Obtain and display the Function URL of the Lambda function. Function URL Retrieval It is a simple but lengthy process to do manually, so I’ve created a script using AWS CLI. Feel free to modify the variables and run it yourself. The script is quite complex; you can do all the steps manually in the GUI if you want, apart from pushing the container. If you are not doing this on an M-series Mac, be sure to change Linux/arm64 to Linux/amd64. #!/bin/bash

# Variables
AWS_REGION="eu-central-1"
ECR_REPO_NAME="llm-lambda"
IMAGE_TAG="latest"
LAMBDA_FUNCTION_NAME="llm-lambda"
LAMBDA_ROLE_NAME="llm-lambda-role" # Role name to create, not ARN
DOCKER_PLATFORM="linux/arm64" # Change as needed, e.g., linux/amd64
IAM_POLICY_FILE="trust-policy.json"
PAGER= # Disable pager for AWS CLI

# Authenticate with AWS
aws configure list # Optional, just to verify AWS CLI is configured

# Check if the ECR repository exists
REPO_EXISTS=$(aws ecr describe-repositories --repository-names $ECR_REPO_NAME --region $AWS_REGION 2>&1)

if [ $? -ne 0 ]; then
    echo "Repository does not exist. Creating repository: $ECR_REPO_NAME"
    # Create ECR repository
    aws ecr create-repository --repository-name $ECR_REPO_NAME --region $AWS_REGION
else
    echo "Repository $ECR_REPO_NAME already exists. Skipping creation."
fi

# Check if the Lambda IAM role exists
ROLE_EXISTS=$(aws iam get-role --role-name $LAMBDA_ROLE_NAME 2>&1)

if [ $? -ne 0 ]; then
    echo "IAM role does not exist. Creating role: $LAMBDA_ROLE_NAME"
    # Create IAM role for Lambda
    aws iam create-role --role-name $LAMBDA_ROLE_NAME --assume-role-policy-document file://$IAM_POLICY_FILE
    aws iam attach-role-policy --role-name $LAMBDA_ROLE_NAME --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
else
    echo "IAM role $LAMBDA_ROLE_NAME already exists. Skipping creation."
fi

# Get login command from ECR and execute it to authenticate Docker to the registry
aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $(aws sts get-caller-identity --query Account --output text).dkr.ecr.$AWS_REGION.amazonaws.com

# Build the Docker image using Docker Compose with specific platform
DOCKER_BUILDKIT=1 docker-compose build --build-arg BUILDPLATFORM=$DOCKER_PLATFORM

# Tag the Docker image for ECR
docker tag llm-lambda:latest $(aws sts get-caller-identity --query Account --output text).dkr.ecr.$AWS_REGION.amazonaws.com/$ECR_REPO_NAME:$IMAGE_TAG

# Push the Docker image to ECR
docker push $(aws sts get-caller-identity --query Account --output text).dkr.ecr.$AWS_REGION.amazonaws.com/$ECR_REPO_NAME:$IMAGE_TAG

# Get the IAM role ARN
LAMBDA_ROLE_ARN=$(aws iam get-role --role-name $LAMBDA_ROLE_NAME --query 'Role.Arn' --output text)

# Check if Lambda function exists
FUNCTION_EXISTS=$(aws lambda get-function --function-name $LAMBDA_FUNCTION_NAME --region $AWS_REGION 2>&1)

# Parameters for Lambda function
LAMBDA_TIMEOUT=300 # 5 minutes in seconds
LAMBDA_MEMORY_SIZE=10240 # Maximum memory size in MB
LAMBDA_IMAGE_URI=$(aws sts get-caller-identity --query Account --output text).dkr.ecr.$AWS_REGION.amazonaws.com/$ECR_REPO_NAME:$IMAGE_TAG

# Deploy or update the Lambda function
if echo $FUNCTION_EXISTS | grep -q "ResourceNotFoundException"; then
    echo "Creating new Lambda function: $LAMBDA_FUNCTION_NAME"
    aws lambda create-function --function-name $LAMBDA_FUNCTION_NAME \
        --region $AWS_REGION \
        --role $LAMBDA_ROLE_ARN \
        --timeout $LAMBDA_TIMEOUT \
        --memory-size $LAMBDA_MEMORY_SIZE \
        --package-type Image \
        --architectures arm64 \
        --code ImageUri=$LAMBDA_IMAGE_URI

    aws lambda create-function-url-config --function-name $LAMBDA_FUNCTION_NAME \
        --auth-type "NONE" --region $AWS_REGION

    # Add permission to allow public access to the Function URL
    aws lambda add-permission --function-name $LAMBDA_FUNCTION_NAME \
        --region $AWS_REGION \
        --statement-id "FunctionURLAllowPublicAccess" \
        --action "lambda:InvokeFunctionUrl" \
        --principal "*" \
        --function-url-auth-type "NONE"
else
    echo "Updating existing Lambda function: $LAMBDA_FUNCTION_NAME"
    aws lambda update-function-code --function-name $LAMBDA_FUNCTION_NAME \
        --region $AWS_REGION \
        --image-uri $LAMBDA_IMAGE_URI
fi

# Retrieve and print the Function URL
FUNCTION_URL=$(aws lambda get-function-url-config --region $AWS_REGION --function-name $LAMBDA_FUNCTION_NAME --query 'FunctionUrl' --output text)
echo "Lambda Function URL: $FUNCTION_URL" Now, let’s add proper permissions and run the script. chmod +x deploy.sh
./deploy.sh
# AWS_PROFILE=your_profile ./deploy.sh 
# if you want to use a different profile than default As a result of the script, you should se the lambda function URL in your terminal. Let’s use it for a test. PROMPT="Create five questions for a job interview for a senior python software engineer position."

curl $LAMBDA_URL -d "{ \"prompt\": \"$PROMPT\" }" \
    | jq -r '.choices[0].text, .usage' Like before, the first run will take longer. chmod +x test_remote.sh
LAMBDA_URL=your_url ./test_remote.sh You should receive back an output similar to the one at the top of the article. If you want to see the full project code, see . github Performance Cost and speed You may have noticed that I chose to run the model under the configuration: 6 core multi-threading 10GB of RAM 5-minute timeout How was the performance, and how much RAM was used? Lambda never reserves compute for you and sometimes loads your program “cold.” When you make a request, anything you need to load (like a model) needs to happen at this moment. We call this a cold start. For the prompt about five interview questions, the execution time after the cold start was 17 seconds, which is not bad at all. After that, it consistently took 9 seconds to generate an output, tested in 4 subsequent runs. The average output length was 83 tokens. So, when running hot. we achieved ~9.2 tokens/second As a comparison, community-reported speeds for OpenAI models are GPT-4 10t/s, GPT-4-turbo 48t/s, and GPT-3-turbo 50-100t/s. So we’re matching a decent but not industry-leading speed with little effort. You could run this prompt 4444 times in a free tier. Above that, . The same 1000 runs with a GPT-3.5 turbo would cost $0.2. This makes this endeavor not really cost-effective, so low cost shouldn’t be your aim if you’re going to implement this in production. it would cost $1.2 per 1000 runs Optimization Could we run the model cheaper? The model uses under 2600MB of memory consistently. We could lower the allocated resources, with a caveat that 6 core CPU is assigned only to Lambdas above ~8846MB. I ran the first test at 3500MB (3 CPU cores/threads) to see how it compares. I didn’t change the code to run only 3 threads, which had a downgrading effect. In this case, the lambda always timed out (ran over 5 minutes, which meant the request was canceled). After changing the number of threads to the proper 3, the execution almost timed out at a cold start, returning results after 4 minutes and 33 seconds; the subsequent 3 runs took 26-40 seconds, averaging 3.15 tokens/second. 1000 paid runs would cost So we got slower and paid more, not a good optimisation! With this in mind, it makes sense to pay for more memory because AWS assigns CPU resources proportionally to the reserved RAM amount. $1.45. Next steps As you can see, running an open-source language model is possible even in a regular environment like AWS Lambda. The cost and quality of output do not match industry-leading services but can serve as a use-case-specific solution. If anything, this was a great opportunity to learn about basic LLM ecosystem libraries and tools, as well as strengthen your AWS knowledge. Some possible next steps if you want to keep exploring and bringing this project closer to production-ready: Try to run a different (bigger) model or a less compressed version of Microsoft Phi-2. Create a CDK config to automatically deploy the model and create resources. Create a GitHub Action to validate the project and run ./deploy.sh. Try a different serverless platform to run the model. Subscribe to my newsletter and follow me on social media. I may write about some of these topics, let me know in the comment if you’re interested in any of them! If you bump into any issues let me know as well. What do you think about this way of deploying LLMs? Subscribe to my profile by filling in your email address on the left, and be up-to-date with my articles! Don't forget to follow me on Twitter and subscribe to my blog’s for more tips and insights! @ horosin_ newsletter If you don't have Twitter, you can also follow me on . LinkedIn Also published . here

Deploying Open-Source Language Models on AWS Lambda

Introduction As LLM technologies gain more mainstream adoption and the ecosystem begins to mature, organizations are beginning to recognize the limits and costs of using LLM technologies. Many enterprises, originally enthusiastic about the application of , have abandoned centralized initiatives, pursuing instead a strategy of encouraging decentralized efforts to incorporate services like ChatGPT and Claude into their workflows. LLM technologies There are several reasons for this phenomenon. Lack of LLM expertise, MLOps requirements, and reliance on specialized GPU infrastructure are all barriers to implementing large-scale AI initiatives. Of these, however, the thorniest issue is the reliance on GPUs. In this article, we will discuss the specific difficulties posed by GPU reliance, explore a potential solution, and look at an exciting example from one of the trailblazing companies working in this area. GPU Availability as Limitations to LLMs Most publicly available and highly performant models, such as GPT-4, Llama 2, and Claude, all rely on highly specialized GPU infrastructure. GPT-4, one of the largest models commercially available, famously runs on a cluster of 8 A100 GPUs. Llama 2’s 70B model, which is much smaller, still requires at least an A40 GPU to run at a reasonable speed. This level of GPU requirement practically forecloses the possibility of running these models locally - a A100 GPU, assuming you can find a seller, costs close to $25,000. Once you obtain the GPUs, you need specialized skills to set up and maintain the servers. Very few organizations would be willing to incur such an outlay to experiment with LLM technologies. To solve this problem, several startups and cloud providers have developed extensive PaaS offerings. Some services like Replicate, which I have used in past articles and projects, allow users to rent GPU servers and pay for compute time used. Other providers, like OpenAI and Anthropic, offer their models as a per-token API, further abstracting away infrastructural complexities. However, these services require data to be sent to an external network, which makes using these services a non-starter for privacy-conscious organizations. Additionally, many of these services suffer from shortages during demand spikes as GPU usage overtakes availability, making them unreliable options for production-critical workloads. Additionally, GPU time, no matter how they are charged for, is expensive for large compute tasks - the companies that own and operate these GPUs need a return on their investment, after all. While these costs are nearly negligible for experimental use cases, commercial use cases often require embedding large contexts, fine-tuning, or multi-shot examples. These costs represent a significant barrier to adoption, especially for organizations with large datasets or those that lack the financial resources of large U.S. firms. In a previous article, we explored parameter compression as one strategy to reduce dependence on GPUs. In today’s article, we will explore another exciting technique called quantization. Before we dive into the exploration, however, we might want to learn a little bit about quantization first. Quantization (Optional Reading) In this section, we will briefly go over the basics of quantization. However, if you’re simply looking for a way to run powerful LLMs locally on your computer, you can feel free to skip this section for now and come back later. LLMWare, the company whose technology we will be using today, has built some amazing tools that let you get started with quantized models without having to get into the nitty-gritty behind it all. C/C++ implementations What is Quantization? Quantization is a technique that seeks to reduce the computational and memory requirements of running an LLM by using lower-precision numerical types. Many popular open-source models, such as Llama, Falcon, and Alpaca, use PyTorch as the underlying framework. By default, PyTorch models use 32-bit floating points, meaning a single parameter takes up 32 “bits” in the GPU memory. Quantization aims to replace these parameters with 16-bit floating points, 8-bit integers, or even 4-bit integers. Successful quantization leads to dramatic improvements in computational speed and reductions in memory usage, meaning large models become runnable on lower-end GPUs, embedded graphic chips, or even CPUs. This idea has been around for a while - PyTorch itself has added support for 16-bit floating points and model compilations as the technology matured, but progress has been slow due to early design decisions in the PyTorch framework. Does Quantization Degrade Performance? At this point, it’s natural to wonder, wouldn’t this severely degrade the accuracy of the model? The short answer is yes, but only if you do it carelessly. Every optimization comes with inherent trade-offs, but with some specialized techniques, researchers have been able to squeeze incredibly stable performance out of highly quantized models. While we won’t go into extreme technical detail, let’s go over the broad strokes of the most common strategies being used right now. If you want to learn more, you can find out more about it in a from HuggingFace. guide Calibrated Quantization During the quantization process, a calibration dataset is run through the model. The value of each parameter is recorded, and the range is used to determine how the parameters get quantized. Assuming the calibration dataset is representative of the inputs that the model will encounter, this results in improved accuracy of the resulting model. Quantization-Aware Whereas Calibrated Quantization happens after training, Quantization-Aware Training tries to optimize the model during training. While the model is training, the activations are put through a “fake quantization,” simulating errors that will likely be introduced by the quantization process. The model is then able to adapt to the errors, resulting in a more robust model that can specifically adapt to the potential distortions. Llama.cpp and GGUF While PyTorch quantization and optimizations have been long blocked by framework design, two recent open-source technologies broke through these barriers and made quantization technologies much more accessible to the general public. Let’s briefly cover them below. Llama.cpp Llama.cpp was a project by Georgi Gerganov to port the Llama model into C/C++. This got rid of the complexity introduced by PyTorch, and the native implementation allowed quantization to be implemented directly. Therefore, the resulting model could run with up to 4-bit integer quantization, allowing high-parameter-count Llama models to be run without a specialized GPU. The project has since been extended by the community to include a roster of open-source models, including popular ones like Falcon and Mistral. GGUF GGUF is Llama.cpp’s file format for storing and transferring model information. Quantized models are stored in this format so that they can be loaded and run by the end-user. GGUF is the successor format to GGML and aims to improve on GGML by providing more extensibility, backward compatibility, and stability while still allowing for rapid development. The development of a universal file format opened the door for the open-source community to extend the Llama.cpp to optimize other models, and innovators like TheBloke and LLMWare have been working over the last several months to miniaturize popular open-source models. LLMWare’s Quantized Dragon Model In today’s example, we will be using open-source libraries and quantized models provided by LLMWare, which provides convenient tools for quickly building specialized RAG workflows. Who is LLMWare? LLMWare, a generative AI company specializing in the legal and financial industries, has been actively involved in the quantization community. As I’ve written before, their focus on privacy-conscious sectors makes them a natural candidate for experimenting with and innovating in miniaturization technologies. Previously, I wrote about their RAG-optimized BLING models that squeeze incredible performance out of 1 to 3 billion parameter models for specialized tasks like contract review and financial analysis. While most open-source models with such parameter counts tend to be only useful for toy problems, LLMWare is able to generate production-ready performance out of these models by training them for narrowly targeted tasks. These miniaturized models are then able to run without an external GPU, allowing for increased privacy and scalability. What is Dragon? Dragon is a collection of LLMs that can be thought of as more powerful versions of their BLING cousins. The original intent of Dragon was to train a higher parameter model using the same instruction fine-tuning techniques, providing an option to users that need more performance and have access to lower-end GPUs. The added parameter count resulted in more powerful models that could leverage larger context windows and generate more complex outputs, but required the user to have more specialized hardware, such as a GPU-embedded laptop or a cloud compute container with a GPU attached. However, they still represented an improvement over the extremely large models, which would require waiting for access to scarce A40 or A100 GPUs. Quantized Dragon, the Best of Both Worlds Given the above, it is easy to see why quantization gave a significant boost to LLMWare’s suite of AI tools. With quantization, a user could run Dragon-tier models on the same environment as BLING models, allowing for much more powerful analysis on commodity computers. Over the course of the last month, LLMWare has published quantized versions of several Dragon models. Today, we will evaluate LLMWare’s Dragon model built on top of Llama with a legal analysis RAG problem and compare it with a similar BLING model. Interested reachers can also explore other models - a Mistral-based model and a Yi-Based model are available from LLMWare at the time of this writing. Additionally, has made running inferences on Llama.cpp models a breeze with their tight integrations with the ctransformers library, which allows gguf models to be swapped seamlessly with PyTorch based models. LLMWare We will use a Macbook Air with M1 Chip for this experiment, meaning we will only be using widely available hardware for this exercise. Testing Out Quantized Dragon Remember that in my previous article, we built a RAG application focused on legislation search. We used vector search to quickly search through several large legislations, found sections relevant to our question about Qualified Opportunity Zone Partnership Interest, and ran the question through a BLING model. In today’s article, we will run the same question through LLMWare’s quantized Dragon model and determine if it performs better than BLING models. In order to focus on model comparison and to reduce the amount of prior knowledge required, we will do a lot of the PDF parsing and vector search manually. This has the added benefit of making the problem artificially harder for the model - LLMWare’s default embedding search chunks the source material to about 1000 tokens, but handling the parsing manually allows us to bump the context up to around 3000 tokens. This will help us clearly demonstrate the difference between the Dragon and BLING models. However, you should be able to easily integrate with the rest of the LLMWare’s ecosystem if you want to leverage their tools by following the setup steps from my last article on LLMWare. In fact, if you simply replace the name of the BLING models with the quantized Dragon model from this article, everything should run seamlessly. Without further ado, let’s get started! First, let’s import the required dependencies: import sklearn
import sklearn.metrics # for cosine similarity
from llmware.prompts import Prompt
import time
import os
from openai import OpenAI
from PyPDF2 import PdfReader 

client = OpenAI() # the library now loads the key automatically as an environment variable. We can now load the PDF. In the previous example, we loaded several large legislations, but for today, we will only focus on the PDF version of the Tax Cuts and Jobs Act of 2017. reader = PdfReader([path to PDF of tax cuts and jobs act]) Now we can generate the embeddings for each page: embeddings = []
for pg in reader.pages:
    text = pg.extract_text() 
    embeddings.append(client.embeddings.create(
        input=text,
        model="text-embedding-ada-002"
    ).data[0].embedding) Let’s also generate the embeddings for the question we’re going to ask: question = 'What is a qualified opportunity zone partnership interest?'
q_embed = client.embeddings.create(
        input=question,
        model="text-embedding-ada-002"
    ).data[0].embedding With the embedding in hand, we can perform a vector search. Because our search space is small, we can just do this manually. cos_sim = [(idx, sklearn.metrics.pairwise.cosine_similarity([e], [q_embed])[0][0]) for idx, e in enumerate(embeddings)] Now we can take the most relevant page (which is index 132 or page 133 if you want to verify the results): most_relevant = sorted(cos_sim, key=lambda x: x[1], reverse=True)[0][0] And with that, we have come to the most crucial step. We will instantiate an LLMWare Prompter object with the quantized Llama Dragon model. The Prompter class is key here because it handles the prompt engineering for us and makes sure that our prompt is consistent with the structure of Dragon’s training data. The prompt class also automatically handles the llamacpp binding, so you can use the quantized Dragon model exactly like other models. model_name = "llmware/dragon-llama-7b-gguf"
prompter = Prompt().load_model(model_name)

response = prompter.prompt_main(question, context='\n\n'.join([reader.pages[132].extract_text()]), 
                                prompt_name="default_with_context", temperature=0.3) Wait a little while, and you should see the function call return. Now print the results: print(response['llm_response']) And you should see something like the following: • A capital or profits interest acquired by the qualified opportunity fund after December 31, 2017, from the partnership solely in exchange for cash;
•As of the time such interest was acquired, the partnership was a qualified opportunity zone business (or, in the case of a new partnership, it was being organized for purposes of being a qualified opportunity zone business);
•During substantially all of the qualified opportunity fund's holding period for such interest, the partnership qualified as a qualified opportunity zone business. This is quite a good answer! For comparison, let’s see how a BLING model would perform on the same problem. One of the issues we can expect is that the large context size can “overwhelm” a lower-parameter model and lead to a less informative answer. In my previous experiments, the sheared llama 2.7b was one of the best performers for this problem, so I decided to use that as the representative of the BLING models. model_name_2 = "llmware/bling-sheared-llama-2.7b-0.1"
prompter2 = Prompt().load_model(model_name_2)

response = prompter2.prompt_main(question, context='\n\n'.join([reader.pages[132].extract_text()]), 
                                prompt_name="default_with_context", temperature=0.3) After some processing, you should see something like this. A qualified opportunity zone partnership interest is a capital or profits interest in a domestic partnership if such interest is acquired by the qualified opportunity fund after December 31, 2017, from the partnership solely in exchange for cash. The response is still good but misses some of the details captured by the Dragon model. Specifically, the answer misses the holding period requirement and the new business case. This conforms to our expectations about the lower-parameter models’ difficulty with processing larger contexts. Interested readers can extend this experiment by using even lower parameter models or increasing the size of the given context. You should see the effect become increasingly more pronounced, after which the model will give a short, garbled answer. From this experiment, it should be clear that quantized Dragon models are able to outperform lower-parameter models for their intended use cases without noticeably compromising the model’s accuracy. And with that, we have used a quantized model to solve a real-world use case and learned about its performance characteristics in the process! Conclusion Today, we explored the exciting field of LLM quantization and looked at how companies like LLMWare are taking advantage of these developments to enhance their specialized language models. As I’ve argued before, miniaturization represents one of the most promising paths to the widespread adoption of AI technologies. By combining specialization, fine-tuning, and quantization, innovators in the AI space can create scalable and performant models that solve real-world problems. You can find LLMWare’s RAG framework on and their DRAGON and BLING models on LLMWare’s . Github Hugging Face repository By the way, I am working on an exciting project that seeks to use language AI and miniaturization to revolutionize education in the developing world. We’re working with incredible activists and educators worldwide, and we’re working to bridge the global digital divide. If you’d like to learn more about my project or simply want to talk about exciting developments in the LLM space, please do not hesitate to reach out to me on either or . Github LinkedIn

Run Llama Without a GPU! Quantized LLM with LLMWare and Quantized Dragon

একটি GPU ছাড়া লামা চালান! এলএলএমওয়্যার এবং কোয়ান্টাইজড ড্রাগনের সাথে কোয়ান্টাইজড এলএলএম

Führen Sie Llama ohne GPU aus! Quantisiertes LLM mit LLMWare und Quantized Dragon

¡Ejecute Llama sin GPU! LLM cuantificado con LLMWare y Quantized Dragon

Exécutez Llama sans GPU ! LLM quantifié avec LLMWare et Quantized Dragon

बिना जीपीयू के लामा चलाएं! एलएलएमवेयर और क्वांटाइज्ड ड्रैगन के साथ क्वांटाइज्ड एलएलएम

GPU なしで Llama を実行しましょう! LLMWare と Quantized Dragon を使用した Quantized LLM

GPU 없이 Llama를 실행해보세요! LLMWare 및 Quantized Dragon을 사용한 양자화 LLM

Execute o Llama sem GPU! LLM Quantizado com LLMWare e Dragão Quantizado

Запустите Llama без графического процессора! Квантованный LLM с LLMWare и Quantized Dragon

Lama'yı GPU olmadan çalıştırın! LLMWare ve Quantized Dragon ile Quantized LLM

Chạy Llama mà không cần GPU! LLM lượng tử hóa với LLMWare và Quantized Dragon

无需 GPU 即可运行 Llama！使用 LLMWare 和 Quantized Dragon 进行量化 LLM

With the rise of Large Language Models and their impressive capabilities, many fancy applications are being built on top of giant LLM providers like OpenAI and Anthropic. The myth behind such applications is the RAG framework, which has been thoroughly explained in the following articles: Building RAG-based LLM Applications for Production Retrieval Augmented Generation (RAG) Explained: Understanding Key Concepts What is retrieval-augmented generation? To become familiar with RAG, I recommend going through these articles. This post, however, will skip the basics and guide you directly on building your own RAG application that can run locally on your laptop without any worries about data privacy and token cost. We will build an application that is something similar to F but simpler. Where users can upload a PDF document and ask questions through a straightforward UI. Our tech stack is super easy with Langchain, Ollama, and Streamlit. ChatPD : The most critical component of this app is the LLM server. Thanks to , we have a robust LLM Server that can be set up locally, even on a laptop. While is an option, I find Ollama, written in Go, easier to set up and run. LLM Server Ollama llama.cpp : Undoubtedly, the two leading libraries in the LLM domain are and . For this project, I’ll be using Langchain due to my familiarity with it from my professional experience. An essential component of any RAG framework is vector storage. We’ll be using here, as it integrates well with Langchain. RAG Langchain LLamIndex Chroma : The user interface is also an important component. Although there are many technologies available, I prefer using , a Python library, for peace of mind. Chat UI Streamlit Okay, let’s start setting it up. Setup Ollama As mentioned above, setting up and running Ollama is straightforward. First, visit and download the app appropriate for your operating system. ollama.ai Next, open your terminal, and execute the following command to pull the latest . While there are many other , I choose Mistral-7B for its compact size and competitive quality. Mistral-7B LLM models available ollama pull mistral Afterward, run to verify if the model was pulled correctly. The terminal output should resemble the following: ollama list Now, if the LLM server is not already running, initiate it with . If you encounter an error message like , it indicates the server is already running by default, and you can proceed to the next step. ollama serve "Error: listen tcp 127.0.0.1:11434: bind: address already in use" Build the RAG Pipeline The second step in our process is to build the RAG pipeline. Given the simplicity of our application, we primarily need two methods: and . ingest ask The method accepts a file path and loads it into vector storage in two steps: first, it splits the document into smaller chunks to accommodate the token limit of the LLM; second, it vectorizes these chunks using Qdrant FastEmbeddings and stores them into Chroma. ingest The method handles user queries. Users can pose a question, and then the RetrievalQAChain retrieves the relevant contexts (document chunks) using vector similarity search techniques. ask With the user's question and the retrieved contexts, we can compose a prompt and request a prediction from the LLM server. from langchain.vectorstores import Chroma
from langchain.chat_models import ChatOllama
from langchain.embeddings import FastEmbedEmbeddings
from langchain.schema.output_parser import StrOutputParser
from langchain.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.schema.runnable import RunnablePassthrough
from langchain.prompts import PromptTemplate
from langchain.vectorstores.utils import filter_complex_metadata


class ChatPDF:
    vector_store = None
    retriever = None
    chain = None

    def __init__(self):
        self.model = ChatOllama(model="mistral")
        self.text_splitter = RecursiveCharacterTextSplitter(chunk_size=1024, chunk_overlap=100)
        self.prompt = PromptTemplate.from_template(
            """
            <s> [INST] You are an assistant for question-answering tasks. Use the following pieces of retrieved context 
            to answer the question. If you don't know the answer, just say that you don't know. Use three sentences
             maximum and keep the answer concise. [/INST] </s> 
            [INST] Question: {question} 
            Context: {context} 
            Answer: [/INST]
            """
        )

    def ingest(self, pdf_file_path: str):
        docs = PyPDFLoader(file_path=pdf_file_path).load()
        chunks = self.text_splitter.split_documents(docs)
        chunks = filter_complex_metadata(chunks)

        vector_store = Chroma.from_documents(documents=chunks, embedding=FastEmbedEmbeddings())
        self.retriever = vector_store.as_retriever(
            search_type="similarity_score_threshold",
            search_kwargs={
                "k": 3,
                "score_threshold": 0.5,
            },
        )

        self.chain = ({"context": self.retriever, "question": RunnablePassthrough()}
                      | self.prompt
                      | self.model
                      | StrOutputParser())

    def ask(self, query: str):
        if not self.chain:
            return "Please, add a PDF document first."

        return self.chain.invoke(query)

    def clear(self):
        self.vector_store = None
        self.retriever = None
        self.chain = None The prompt is sourced from the Langchain hub: . This prompt has been tested and downloaded thousands of times, serving as a reliable resource for learning about LLM prompting techniques. Langchain RAG Prompt for Mistral You can learn more about LLM prompting techniques . here More details on the implementation: : We use PyPDFLoader to load the PDF file uploaded by the user. The RecursiveCharacterSplitter, provided by Langchain, then splits this PDF into smaller chunks. It's important to filter out complex metadata not supported by ChromaDB using the function from Langchain. ingest filter_complex_metadata For vector storage, Chroma is used, coupled with as our embedding model. This lightweight model is then transformed into a retriever with a score threshold of 0.5 and k=3, meaning it returns the top 3 chunks with the highest scores above 0.5. Finally, we construct a simple conversation chain using . Qdrant FastEmbed LECL : This method simply passes the user's question into our predefined chain and then returns the result. ask : This method is used to clear the previous chat session and storage when a new PDF file is uploaded. clear Draft A Simple UI For a simple user interface, we will use , a UI framework designed for the fast prototyping of AI/ML applications. Streamlit import os
import tempfile
import streamlit as st
from streamlit_chat import message
from rag import ChatPDF

st.set_page_config(page_title="ChatPDF")


def display_messages():
    st.subheader("Chat")
    for i, (msg, is_user) in enumerate(st.session_state["messages"]):
        message(msg, is_user=is_user, key=str(i))
    st.session_state["thinking_spinner"] = st.empty()


def process_input():
    if st.session_state["user_input"] and len(st.session_state["user_input"].strip()) > 0:
        user_text = st.session_state["user_input"].strip()
        with st.session_state["thinking_spinner"], st.spinner(f"Thinking"):
            agent_text = st.session_state["assistant"].ask(user_text)

        st.session_state["messages"].append((user_text, True))
        st.session_state["messages"].append((agent_text, False))


def read_and_save_file():
    st.session_state["assistant"].clear()
    st.session_state["messages"] = []
    st.session_state["user_input"] = ""

    for file in st.session_state["file_uploader"]:
        with tempfile.NamedTemporaryFile(delete=False) as tf:
            tf.write(file.getbuffer())
            file_path = tf.name

        with st.session_state["ingestion_spinner"], st.spinner(f"Ingesting {file.name}"):
            st.session_state["assistant"].ingest(file_path)
        os.remove(file_path)


def page():
    if len(st.session_state) == 0:
        st.session_state["messages"] = []
        st.session_state["assistant"] = ChatPDF()

    st.header("ChatPDF")

    st.subheader("Upload a document")
    st.file_uploader(
        "Upload document",
        type=["pdf"],
        key="file_uploader",
        on_change=read_and_save_file,
        label_visibility="collapsed",
        accept_multiple_files=True,
    )

    st.session_state["ingestion_spinner"] = st.empty()

    display_messages()
    st.text_input("Message", key="user_input", on_change=process_input)


if __name__ == "__main__":
    page() Run this code with the command to see what it looks like. streamlit run app.py Okay, that’s it! We now have a ChatPDF application that runs entirely on your laptop. Since this post mainly focuses on providing a high-level overview of how to build your own RAG application, there are several aspects that need fine-tuning. You may consider the following suggestions to enhance your app and further develop your skills: : Currently, it doesn’t remember the conversation flow. Adding temporary memory will help your assistant be aware of the context. Add Memory to the Conversation Chain : it’s okay to chat about one document at a time. But imagine if we could chat about multiple documents — you could put your whole bookshelf in there. That would be super cool! Allow multiple file uploads : While Mistral is effective, there are many other alternatives available. You might find a model that better fits your needs, like LlamaCode for developers. However, remember that the choice of model depends on your hardware, especially the amount of RAM you have 💵 Use Other LLM Models : There’s room for experimentation within RAG. You might want to change the retrieval metric, the embedding model,.. or add layers like a re-ranker to improve results. Enhance the RAG Pipeline Finally, thank you for reading. If you find this information useful, please consider subscribing to my or my personal . I plan to write more about RAG and LLM applications, and you’re welcome to suggest topics by leaving a comment below. Cheers! Substack blog Full source code: https://github.com/vndee/local-rag-example

A Tutorial On How to Build Your Own RAG and How to Run It Locally: Langchain + Ollama + Streamlit

কীভাবে আপনার নিজের RAG তৈরি করবেন এবং কীভাবে এটি স্থানীয়ভাবে চালাবেন তার একটি টিউটোরিয়াল: ল্যাংচেইন + ওল্লামা + স্ট্রিমলিট

Ein Tutorial zum Erstellen Ihres eigenen RAG und zum lokalen Ausführen: Langchain + Ollama + Streamlit

Un tutorial sobre cómo construir su propio RAG y cómo ejecutarlo localmente: Langchain + Ollama + Streamlit

Un tutoriel sur la façon de créer votre propre RAG et de l'exécuter localement : Langchain + Ollama + Streamlit

अपना खुद का आरएजी कैसे बनाएं और इसे स्थानीय स्तर पर कैसे चलाएं, इस पर एक ट्यूटोरियल: लैंगचैन + ओलामा + स्ट्रीमलिट

独自の RAG を構築する方法とローカルで実行する方法に関するチュートリアル: Langchain + Ollama + Streamlit

자신만의 RAG를 구축하고 로컬에서 실행하는 방법에 대한 튜토리얼: Langchain + Ollama + Streamlit

Um tutorial sobre como construir seu próprio RAG e como executá-lo localmente: Langchain + Ollama + Streamlit

Учебное пособие о том, как создать свой собственный RAG и запустить его локально: Langchain + Ollama + Streamlit

Kendi RAG'ınızı Nasıl Oluşturacağınıza ve Yerel Olarak Nasıl Çalıştıracağınıza İlişkin Bir Eğitim: Langchain + Ollama + Streamlit

Hướng dẫn về cách xây dựng RAG của riêng bạn và cách chạy nó cục bộ: Langchain + Ollama + Streamlit

关于如何构建自己的 RAG 以及如何在本地运行它的教程：Langchain + Ollama + Streamlit

Deploying multiple local Ai agents using local LLMs like Llama2 and Mistral-7b. “Never Send A Human To Do A Machine’s Job” — Agent Smith Are you searching for a way to build a whole army of organized ai agents with Autogen using local LLMs instead of the paid OpenAi? Then you came to the right place! Chat LLMs are cool, but giving taking action as an intelligent agent is next level. What about many of them? Meet Microsoft’s latest project. Autogen But there is a problem. Autogen was built to be hooked to OpenAi by default, wich is limiting, expensive and censored/non-sentient. That’s why using a simple LLM locally like is the best way to go. You can also use with any other model of your choice such as , , , , the sky (your hardware) is really the limit. Mistral-7B Llama2 Falcon Vicuna Alpaca The secret is to use openai JSON style of output in your local LLM server such as Oobabooga’s text-generation-webui, then hook it to autogen. That’s what we’re building today. Note there are other methods for making llms spit text in openai apis format as well like the llama.cpp python bindings. 0. Getting Oobabooga’s text-generation-webui, an LLM (Mistral-7b) and Autogen In this tutorial we will be: Setting up OpenAi format extension on Oobabooga Starting the local LLM server with the OpenAi format Hooking it to Autogen Let’s get started! 0. Getting Oobabooga’s Text-Generation-Webui, an LLM (Mistral-7b) and Autogen Before proceeding, it’s recommended to use a virtual environment when installing pip packages. Make a new one and activate it if you feel like. This is a well known program for hosting LLMs in your local machine. Head to and follow the installation guide. It is very straight forward to get started. You might also want to download if you are using an NVIDIA gpu for acceleration. Getting Obbabooga’s Text Generation Webui: text-generation-webui’ s page CUDA After downloading the text generation webui, don’t start it just yet. We need to get a LLM to give life to our agents. Getting an LLM (Mistral-7b-Instruct): Today we’ll be exploring , specifically , an optimized version of the model by TheBloke. You can choose the optimized model perfect for your machine based on the . Mistral-7B Mistral-7B-instruct-v0.1.Q4_K_S.gguf explanation in the description You can choose smaller or bigger models depending on your hardware. Don’t be too scared to try out things in your computer tho, we’re making science here. Head to the , and grab the following: Files and Versions page config.json (will run well in most mid setups) Mistral-7B-instruct-v0.1.Q4_K_S.gguf Once downloaded, head to the text-generation-webui installation folder, and inside it . In there, with the name of your model (or any name you want), like . The path will be like this: open the models folder create a new folder “mistral-7b-instruct” C:/.../text-generation-webui/models/mistral-7b-instruct Place both the file and the in the new folder. config.json model.gguf : To install Microsoft’s multi-agent making python library just install it using the pip package installer in your terminal. Getting Autogen pip install pyautogen 1. Setting Up OpenAi Format Extension on Oobabooga With your brand new text-generation-webui installed and the LLM downloaded, we can proceed on making your local Oobabooga server speak in OpenAi JSON format. You can learn more about OpenAi APIs formats and features it in their . documentation To hook Autogen with our local server, we will need to activate the “openai” extension in the Ooobaboga’s text-generation-webui extensions folder. In your terminal head to folder and in there install its requirements: “text-generation-webui/extensions/openai” pip install -r requirements.txt 2. Starting The Local LLM Server In OpenAi Format Now head back to the root folder in your terminal. Its time to get this baby up and running. /text-generation-webui As the name says, it was meant to be used as a webui, but you can also just keep it running as a server to query apis from other programs you make. To boot it as a local server and with the openai api extension, use the following command according to your current OS. (In my case I named the folder **“**mistral-7b-instruct”) Don’t forget to change the “model” parameter to the folder name we created earlier at /models. Windows: ./start_windows.bat --extensions openai --listen --loader llama.cpp --model mistral-7b-instruct Linux: ./start_linux.sh --extensions openai --listen --loader llama.cpp --model mistral-7b-instruct MacOS: ./start_macos.sh --extensions openai --listen --loader llama.cpp --model mistral-7b-instruct We pass the parameter to load the extension, to start a server we can query from autogen, and wich specify the loader for the model and the model folder name we created earlier, with the config.json and the model.gguf files. extensions openai listen loader model If everything goes right, you might see something like this: The webui is running on your localhost port 7860 as an usual start, but note our OpenAI compatible api is also ready to be used by Autogen at our local host at . http://127.0.0.1:5001/v1 3. Hooking it to Autogen At this point, you already have the autogen lib installed, so it’s time to import it and plug our LLM server. Let’s start with something simple, a single agent interacting with a human (you). Create a new directory wherever you like, and add a new file there. You can also rename the file as you wish. autogen.py Generally to simply connect to OpenAi GPT’s API you would start the file like this: import autogen #start importing the autogen lib

config_list = [
    {
        'model': 'gpt-3.5-turbo',
        'api_key': 'your openai real key here'

    }
] But to use our running local server, we initiate it like this: import autogen #start importing the autogen lib

config_list = [
    {
        "model": "mistral-instruct-7b", #the name of your running model
        "api_base": "http://127.0.0.1:5001/v1", #the local address of the api
        "api_type": "open_ai",
        "api_key": "sk-111111111111111111111111111111111111111111111111", # just a placeholder
    }
] As you don’t need a real key for working locally, we are just using the sk-1111… placeholder. Next, we can setup the agent and the human user. Read the comments for better understanding. import autogen #start importing the autogen lib

config_list = [
    {
        "model": "mistral-instruct-7b", #the name of your running model
        "api_base": "http://127.0.0.1:5001/v1", #the local address of the api
        "api_type": "open_ai",
        "api_key": "sk-111111111111111111111111111111111111111111111111", # just a placeholder
    }
]

# create an ai AssistantAgent named "assistant"
assistant = autogen.AssistantAgent(
    name="assistant",
    llm_config={
        "seed": 42,  # seed for caching and reproducibility
        "config_list": config_list,  # a list of OpenAI API configurations
        "temperature": 0,  # temperature for sampling
        "request_timeout": 400, # timeout
    },  # configuration for autogen's enhanced inference API which is compatible with OpenAI API
)

# create a human UserProxyAgent instance named "user_proxy"
user_proxy = autogen.UserProxyAgent(
    name="user_proxy",
    human_input_mode="NEVER",
    max_consecutive_auto_reply=10, 
    is_termination_msg=lambda x: x.get("content", "").rstrip().endswith("TERMINATE"),
    code_execution_config={
        "work_dir": "agents-workspace", # set the working directory for the agents to create files and execute
        "use_docker": False,  # set to True or image name like "python:3" to use docker
    },
)

# the assistant receives a message from the user_proxy, which contains the task description
user_proxy.initiate_chat(
    assistant,
    message="""Create a posting schedule with captions in instagram for a week and store it in a .csv file.""",
) Remember to change message=”…” with your initial orders. If you just run the script with the message, you may see a new directory called “agents-workspace” with a .csv file in there, created “manually” by the agent. Now let’s go for something a bit more advanced. Multiple agents with roles and contexts. This will work like a “chat group” like any messaging app you know. Their contexts (system message) will tell them how to behave, and wich hierarchy they should obey. This time we will have: Two humans: the admin and the executor. Four agents: the engineer, the scientist, the planner and the critic. import autogen

#Use the local LLM server same as before
config_list = [
    {
        "model": "mistral-instruct-7b", #the name of your running model
        "api_base": "http://127.0.0.1:5001/v1", #the local address of the api
        "api_type": "open_ai",
        "api_key": "sk-111111111111111111111111111111111111111111111111", # just a placeholder
    }
]

# set a "universal" config for the agents
agent_config = {
    "seed": 42,  # change the seed for different trials
    "temperature": 0,
    "config_list": config_list,
    "request_timeout": 120,
}

# humans
user_proxy = autogen.UserProxyAgent(
   name="Admin",
   system_message="A human admin. Interact with the planner to discuss the plan. Plan execution needs to be approved by this admin.",
   code_execution_config=False,
)

executor = autogen.UserProxyAgent(
    name="Executor",
    system_message="Executor. Execute the code written by the engineer and report the result.",
    human_input_mode="NEVER",
    code_execution_config={"last_n_messages": 3, "work_dir": "paper"},
)

# agents
engineer = autogen.AssistantAgent(
    name="Engineer",
    llm_config=agent_config,
    system_message='''Engineer. You follow an approved plan. You write python/shell code to solve tasks. Wrap the code in a code block that specifies the script type. The user can't modify your code. So do not suggest incomplete code which requires others to modify. Don't use a code block if it's not intended to be executed by the executor.
Don't include multiple code blocks in one response. Do not ask others to copy and paste the result. Check the execution result returned by the executor.
If the result indicates there is an error, fix the error and output the code again. Suggest the full code instead of partial code or code changes. If the error can't be fixed or if the task is not solved even after the code is executed successfully, analyze the problem, revisit your assumption, collect additional info you need, and think of a different approach to try.
''',
)

scientist = autogen.AssistantAgent(
    name="Scientist",
    llm_config=agent_config,
    system_message="""Scientist. You follow an approved plan. You are able to categorize papers after seeing their abstracts printed. You don't write code."""
)

planner = autogen.AssistantAgent(
    name="Planner",
    system_message='''Planner. Suggest a plan. Revise the plan based on feedback from admin and critic, until admin approval.
The plan may involve an engineer who can write code and a scientist who doesn't write code.
Explain the plan first. Be clear which step is performed by an engineer, and which step is performed by a scientist.
''',
    llm_config=agent_config,
)

critic = autogen.AssistantAgent(
    name="Critic",
    system_message="Critic. Double check plan, claims, code from other agents and provide feedback. Check whether the plan includes adding verifiable info such as source URL.",
    llm_config=agent_config,
)

# start the "group chat" between agents and humans
groupchat = autogen.GroupChat(agents=[user_proxy, engineer, scientist, planner, executor, critic], messages=[], max_round=50)
manager = autogen.GroupChatManager(groupchat=groupchat, llm_config=agent_config)

# Start the Chat!
user_proxy.initiate_chat(
    manager,
    message="""
find papers on LLM applications from arxiv in the last week, create a markdown table of different domains.
""",
)

# to followup of the previous question, use:
# user_proxy.send(
#     recipient=assistant,
#     message="""your followup response here""",
# ) There you go, you have your new army of agents. I strongly recommend going deeper in the Autogen documentation to understand what else this kind of agency automation is able to do. Also, after understanding how autogen works under the hood you may want to use it via a interface like , or maybe create your own in your company’s dashboard. autogen-ui Now it’s up to you. Orchestrate agents untied from OpenAi limitations, to build a better future for us humans. Always remember that with great power comes great responsibility. So, what are you building next? This post was completely written by a human™ Also published . here

Beep Beep Bop Bop: How to Deploy Multiple AI Agents Using Local LLMs

বিপ বিপ বপ বপ: স্থানীয় এলএলএম ব্যবহার করে একাধিক এআই এজেন্ট কীভাবে মোতায়েন করবেন

Beep Beep Bop Bop: So stellen Sie mehrere KI-Agenten mithilfe lokaler LLMs bereit

Beep Beep Bop Bop: Cómo implementar múltiples agentes de IA utilizando LLM locales

Beep Beep Bop Bop : Comment déployer plusieurs agents IA à l'aide de LLM locaux

बीप बीप बोप बोप: स्थानीय एलएलएम का उपयोग करके एकाधिक एआई एजेंटों को कैसे तैनात करें

Beep Beep Bop Bop: ローカル LLM を使用して複数の AI エージェントをデプロイする方法

Beep Beep Bop Bop: 로컬 LLM을 사용하여 여러 AI 에이전트를 배포하는 방법

Beep Beep Bop Bop: como implantar vários agentes de IA usando LLMs locais

Beep Beep Bop Bop: как развернуть несколько агентов ИИ с помощью локальных LLM

Beep Beep Bop Bop: Yerel LLM'leri Kullanarak Birden Fazla Yapay Zeka Aracısı Nasıl Dağıtılır

Beep Beep Bop Bop: Cách triển khai nhiều tác nhân AI bằng LLM cục bộ

Beep Beep Bop Bop：如何使用本地 LLM 部署多个 AI 代理

alpaca-lora: Experimenting With a Home-Cooked Large Language Model

alpaca-lora: বাড়িতে রান্না করা বড় ভাষার মডেলের সাথে পরীক্ষা করা

alpaca-lora: Experimentieren mit selbstgekochtem großem Sprachmodell

alpaca-lora: experimentando con un modelo de lenguaje grande casero

alpaca-lora : Expérimentation d'un grand modèle de langage fait maison

अल्पाका-लोरा: घर में पकाए गए बड़े भाषा मॉडल के साथ प्रयोग

alpaca-lora: 手作りの大規模言語モデルの実験

alpaca-lora: 집에서 만든 대형 언어 모델 실험

alpaca-lora: experimentando um modelo de linguagem grande feito em casa

alpaca-lora: экспериментируем с домашней моделью большого языка

alpaca-lora: Evde Pişirilmiş Büyük Dil Modeli ile Denemeler

alpaca-lora: Thử nghiệm mô hình ngôn ngữ lớn nấu tại nhà

alpaca-lora：尝试自制的大型语言模型

The Release of Meta’s LLAMA-2 Marks A New Era in The History of Humanity

Everyone is these days, and some of us are poorer than others. So my mission is to fine-tune a model with only one GPU on Google Colab and run the trained model on my laptop using llama.cpp. GPU-poor LLaMA-2 Why fine-tune an existing LLM? A lot has been said about when to do , when to do RAG (Retrieval Augmented Generation), and when to fine-tune an existing LLM model. I will not get into details about those arguments and will leave you with two in-depth analyses to explore on your own. prompt engineering i DeepLearning.ai course by Lamin Blog post “Why You (Probably) Don’t Need to Fine-tune an LLM” by Jessica Yao Assuming you still want to fine-tune your own LLM, let’s get started with fine-tuning. Why be a cheapskate? You can , which has become increasingly affordable both for fine-tuning as well as for inference. There are a few reasons you don’t want to do that: your training data is super secret, you don’t want to pay OpenAI every time you use the fine-tuned model, and you need to use your model without the Internet. In that case, we will use open-source LLMs. fine-tune OpenAI’s GPT-3.5-turbo model Right now, is the golden standard of open-source LLM with good performance and permissible license terms. And we will start with the smallest since it will be cheaper and faster to fine-tune. Once you have gone through the whole process, you will be well on your way to and models if you like. Meta’s LLaMA-2 7B model 13B 70B : Training dataset Dolly 15K by DataBricks : The easiest to use is Google Colab. I believe you do need to have a Colab Pro account which is $10 a month for 100 compute units. In the following examples, you will consume between 20–90 compute units which translates to . I hope we all can afford it, even for cheapskates. Training GPU $2–9 Fine-Tuning LLaMA-2 With QLoRA on a Single GPU We have all heard about the tremendous cost associated with training a large language model, which is not something the average Jack or Jill will undertake. But what we can do is freeze the model weights in an existing LLM (e.g. 7B parameters), while fine-tuning a tiny adapter ( of total parameters, 130M for example). less than 1% One of these adapters is called (Low-Rank Adaptation), not to be confused with the red-haired heroine in the movie “Run, Lola, run!”. LoRA In addition, uses a frozen, 4-bit quantized pre-trained language model instead of a 16-bit model into Low-Rank Adapters (LoRA). Thus we can fit the entire training into the GRAM of a single commodity GPU. QLoRA You can find out the trade-offs between our method and the traditional full-parameter method: Blog post “ Fine-Tuning LLMs: LoRA or Full-Parameter? An in-depth Analysis with Llama 2” There are good tutorials and notebooks on fine-tuning LLaMA-2 models with LoRA, for example: OVH cloud Tutorial and notebook Phillip Schimd Tutorial and notebook In this article, I’m using the with minor changes to the training parameters. OVH Cloud guide I used instance, and the total fine-tuning ran about 7 hours and consumed 91 compute units. Google Colab Pro’s Nvidia A100 high memory Google Colab A100 high memory Nvidia A100 high memory CPU RAM: 83.5GB GPU RAM: 40GB 13 computer units per hour Actual memory usage during training: CPU: 6.1GB (up to, varies) GPU: 25.8GB (up to, varies) You can certainly use a single (15GB GRAM) instance, which will take longer but cost less. I started but did not run through the entire training process, but it was estimated to be about 24 hours and 50 compute units. I’m quite sure someone can use (24GB GRAM) or equivalent consumer GPU for this fine-tuning task as well. T4 high memory Nvidia’s 4090 Note: Phillip Schmid’s script has more tricks to reduce training time: my test run on an A100 high memory instance lasted about an hour and 15 minutes and cost less than 20 compute units. Once the training is done, we to mounted Google Drive so we don’t lose them once the Google Colab session is over: save the LoRA adapter’s final checkpoints output_dir = "results/llama2/final_checkpoint" train(model, tokenizer, dataset, output_dir) You can see that the file “adapter_model.bin” is tiny (152.7B) compared to llama2–7b’s “consolidated.00.pth” (13.5GB). Inference with llama.cpp Both fine-tuning tutorials use GPU-based inference, but a true cheapskate would probably want to use his/her own laptop with a low-spec CPU and GPU. Thus comes into play. Your fine-tuned will run comfortably with fast speed on an M1-based Macbook Pro with 16G unified RAM. You can push to run the 13B model as well if you free up some memory from resource-hungry apps. llama.cpp 7B model There are to get your recently fine-tuned model ready for llama.cpp use. All the models reside in the directory “models”. Let’s create a new directory called “lora” under “models”, copy over all the original llama2–7B files, and then copy over the two adapter files from the previous step. The folder “lora” should have the following files a few simple steps : Convert LoRA adapter model to ggml compatible mode: Step 1 python3 convert-lora-to-ggml.py models/lora : Convert into f16/f32 models: Step 2 python3 convert.py models/lora : Quantize to 4 bits: Step 3 ./quantize ./models/lora/ggml-model-f16.gguf ./models/lora/ggml-model-q4_0.gguf q4_0 Now finally, you have your shining new gguf file that is baked with your special training data. It’s time to use it, or in fancy words ! “inference with it” ./main -m models/lora/ggml-model-q4_0.gguf --color -ins -n -1 , while I have dozens of tabs open in two Chrome browsers, a engine running database and web server, Visual Studio Code, and all the instant messaging systems imaginable all on an average Macbook Pro M1 with 16GB memory. You can see llama-2–7b-Lora is running blazing fast Docker Next steps ! You have just fine-tuned your first personal LLM and run it on your laptop. Now there are : Congratulations a few things you can do next where fine-tuning an existing LLM will give you a unique advantage Define a use case for fine-tuning purposes: typically it will be like DataBricks-dolly-15k in question—answer pairs. Your private data most likely won’t be like that. So you can either use your own scripts, manual labor, and/or GPT-4 to format your data into the right training set. Prepare your own dataset , and compare the different approaches (prompt engineering, RAG, GPT-3.5 fine-tuning, open source LLM fine-tuning). Start with a small amount of training data, and build on your experience and success. Define your evaluation metrics Show the world (and your boss) what you have just built! Feature image created with QLoRA and llama.cpp

The Cheapskate’s Guide to Fine-Tuning LLaMA-2 and Running It on Your Laptop

Natural Language Processing (NLP) is a rapidly evolving field, and Large Language Models (LLMs) are playing a significant role in shaping how we interact with human language. However, the creation of LLMs is often considered the domain of machine learning experts, leaving software engineers to take a back seat. This article challenges this idea and highlights the critical role that software engineers with strong programming skills can play in driving LLMs' growth and innovation, even without in-depth knowledge of machine learning. The Power of LLMs in NLP Before we dive into how software engineers can make their mark in LLM development, let's take a moment to appreciate these models' profound impact on NLP. Beyond just deciphering complex language structures, LLMs have simplified intricate processes in various domains. They have empowered chatbots to provide exceptional customer support, revolutionized language translation, automated text summarization, and refined sentiment analysis. In doing so, they've replaced cumbersome pipelines with elegant, data-driven solutions. Now, let's explore how software engineers can actively contribute: Creating Pet Projects: You don't need a machine learning PhD to embark on exciting LLM projects. Consider the possibilities of generating personalized workout plans, crafting creative writing prompts, or developing unique conversational agents. The scope is boundless. By initiating and sharing these projects, you not only showcase the versatility of LLMs but also expand the realm of practical applications. These projects serve as a bridge between software engineering and NLP, demonstrating the tangible impact of LLMs. For some inspiration, check out these project ideas: AutoGPT and LangChain: Building the Ultimate LLM App How to Build an AI-Powered Personal Assistant App Developing a Travel AI App with ChatGPT Elevating Open Source Visualization Tools Visualization  is pivotal component of the LLM ecosystem. Here, software engineers can contribute to the development and enhancement of open-source visualization tools tailored for LLMs. These tools serve as essential aids in comprehending model behavior and results. By actively participating in the improvement of such visualization tools, frontend software engineers can make LLMs more accessible and user-friendly for a wider audience. Some ideas for initial contribution: - TensorBoard, A web-based tool for visualizing TensorFlow runs and graphs. tensorboard - a viewer for neural network, deep learning, and machine learning models . Netron supports a variety of popular model formats such as TensorFlow, PyTorch, ONNX, Keras, and Caffe2. netron - a model-agnostic visual debugging tool for machine learning . Manifold allows you to explore your model’s behavior by visualizing its decision boundaries in high-dimensional space manifold Reimplementation Challenges Recreating established LLM models from scratch offers a valuable learning journey. While it may not be intended for production use, this exercise provides invaluable insights into the inner workings of models like GPT and BERT. Through this hands-on approach, software engineers can better understand LLMs and explore their capabilities firsthand. Explore these reimplementation challenges: : A C/C++ port of Facebook's LLaMA model on your own laptop with decent performance LLaMA.cpp : A Rust implementation of OpenAI's Whisper model using the Burn framework. Whisper-Burn Developing Model Converters and Wrappers Another practical avenue for contribution is crafting model converters for popular programming languages. Software engineers can make a significant contribution by developing model converters and wrappers for different programming languages. These tools simplify the integration of LLMs into various projects, making them more accessible to a wider audience. By creating effective bindings for languages like Go or .NET, software engineers bridge the gap for developers who prefer those languages. Consider these converter and wrapper projects: : A .NET library granting access to the core of PyTorch. TorchSharp : A library designed primarily for running inference against serialized models from Python versions of PyTorch. go-torch : An open-source format for AI models spanning deep learning and traditional machine learning. ONNX : A library for crafting and training machine learning models in JavaScript, deployable in browsers or on Node.js. TensorFlow.js Conclusion Software engineers bring a unique perspective to the development of large language models. While expertise in machine learning is undoubtedly valuable, programming skills and creativity can also drive progress. Through the act of initiating projects, enhancing open-source visualization tools, reimagining model structures, and developing essential converters and wrappers, software engineers actively contribute to the evolution of LLMs. This collaborative synergy, where diverse skill sets converge, fosters inclusivity, accessibility, and innovation within the field. In essence, whether you're a seasoned software engineer or just embarking on this journey, remember that your role in shaping the future of LLMs is significant. By marrying the power of technology with the ingenuity of software engineering, we collectively redefine the boundaries of what's possible in NLP.

A List of Projects Software Engineers Should Undertake to Learn More About LLMs

My Experience With LLama2 Both as a Developer and a Hacker

একজন বিকাশকারী এবং হ্যাকার উভয় হিসাবে LLama2 এর সাথে আমার অভিজ্ঞতা

Meine Erfahrung mit LLama2 sowohl als Entwickler als auch als Hacker

Mi experiencia con LLama2 como desarrollador y hacker

Mon expérience avec LLama2 en tant que développeur et pirate informatique

एक डेवलपर और हैकर दोनों के रूप में LLama2 के साथ मेरा अनुभव

開発者とハッカーの両方としての LLama2 の経験

개발자이자 해커로서 LLama2에 대한 나의 경험

Minha experiência com LLama2 como desenvolvedor e hacker

Hem Geliştirici hem de Hacker Olarak LLama2 Deneyimim

Trải nghiệm của tôi với LLama2 với tư cách là Nhà phát triển và Hacker

我作为开发者和黑客的 LLama2 体验

Hello everyone! In the previous articles ( and ), I described my experience with the development of an alerting and state checking system for a service operating on a remote server, with communication occurring through a Telegram bot. This way of communication is convenient because the phone with Telegram is always at hand, and sometimes it's even too much effort to take out the laptop when you can quickly check everything in Telegram. part 1 part 2 In this part, I will describe the process of regularly collecting data and forming graphs on the functioning of the service, which can also be obtained through the bot. After a couple of days of using the service described in previous articles, I felt the need to also get graphs directly in Telegram, rather than going for the laptop, opening csv files in Jupyter notebook, etc. And I thought, why not do it? I had doubts that the graphs would not look very good if created blindly, automatically. For instance, labels might overlap with lines or axes. But in the end, using the Seaborn library instead of the usual Matplotlib, without any special settings, it is possible to achieve quite decent graphs. Let's start as usual with creating a virtual environment: cd ~
virtualenv -p python3.8 up_env 
source ~/up_env/bin/activate and installing the necessary dependencies: pip install python-telegram-bot
pip install "python-telegram-bot[job-queue]" --pre
pip install --upgrade python-telegram-bot==13.6.0 
pip install numpy 
pip install seaborn

pip install web3 # add here anything you need As in the previous article, the file does not undergo any changes in this case and remains the same: functions.py import numpy as np
import multiprocessing

from web3 import Web3 #  add those libraries needed for your task

# Helper function that checks a single node
def get_last_block_once(rpc):
    try:
        w3 = Web3(Web3.HTTPProvider(rpc))
        block_number = w3.eth.block_number
        if isinstance(block_number, int):
            return block_number
        else:
            return None
    except Exception as e:
        print(f'{rpc} - {repr(e)}')
        return None


# Main function to check the status of the service that will be called
def check_service():
    # pre-prepared list of reference nodes
    # for any network, it can be found on the website https://chainlist.org/
    list_of_public_nodes = [
        'https://polygon.llamarpc.com',
        'https://polygon.rpc.blxrbdn.com',
        'https://polygon.blockpi.network/v1/rpc/public',
        'https://polygon-mainnet.public.blastapi.io',
        'https://rpc-mainnet.matic.quiknode.pro',
        'https://polygon-bor.publicnode.com',
        'https://poly-rpc.gateway.pokt.network',
        'https://rpc.ankr.com/polygon',
        'https://polygon-rpc.com'
    ]
    
    # parallel processing of requests to all nodes
    with multiprocessing.Pool(processes=len(list_of_public_nodes)) as pool:
        results = pool.map(get_last_block_once, list_of_public_nodes)
        last_blocks = [b for b in results if b is not None and isinstance(b, int)]

    # define the maximum and median value of the current block
    med_val = int(np.median(last_blocks))
    max_val = int(np.max(last_blocks))
    # determine the number of nodes with the maximum and median value
    med_support = np.sum([1 for x in last_blocks if x == med_val])
    max_support = np.sum([1 for x in last_blocks if x == max_val])

    return max_val, max_support, med_val, med_support A significant difference from the two previous scripts to launch the bot is that to create a graph and then send it through the bot, you first need to collect data, or rather, collect it at some regularity. Since, unlike alerting, here I want the data to be written regardless of the operability of the bot itself (for example, downtime during bot updates), the script for data collection will be put into a separate file and will be run through cron at some regularity. So, the code for the data collection script : data_collection.py import datetime 
import csv

# Import necessary functions
from functions import get_last_block_once, check_service

# File path for logging
LOG_FILE = '../logs.csv'
# Address of the node whose state I am monitoring (public node in this case)
OBJECT_OF_CHECKING = 'https://polygon-mainnet.chainstacklabs.com'

# Function to save one measurement to a CSV file
def save_log(log_data):
    with open(LOG_FILE, mode='a', newline='') as log_file:
        log_writer = csv.writer(log_file)
        log_writer.writerow(log_data)


if __name__ == '__main__':
    # Call the main function for checking the network state
    max_val, max_support, med_val, med_support = check_service()
    # Call the function for checking the state of the node being monitored
    last_block = get_last_block_once(OBJECT_OF_CHECKING)

    # Current date and time
    timestamp_string = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")

    # Formulate data string and save to file
    log_data = [timestamp_string, max_val, max_support, med_val, med_support, last_block]
    save_log(log_data) This script is run through cron with a regularity, for example, every 5 minutes. To open cron settings: crontab -e In the settings file, we save the line to run the script in the created environment, and separately log errors in another file: up_env * * * * *  cd ~; source up_env/bin/activate; cd /path/to/script; python data_collection.py >> ~/collect.log 2>&1 As stated in the file, data will be saved in the file. We will use them to build graphs. Now let's move on to the script describing the work of the Telegram bot. Import the necessary dependencies and set the constants: data_collection.py ../logs.csv import telegram 
from telegram.ext import Updater, CommandHandler, Filters

# For reading data and building charts
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

import datetime
import io

# Here I can specify a limited circle of bot users by listing their usernames
ALLOWED_USERS = ['your_telegram_account', 'someone_else']
# The node address whose status I am tracking (also a public node in this case)
OBJECT_OF_CHECKING = 'https://polygon-mainnet.chainstacklabs.com'
# Threshold for highlighting critical lag
THRESHOLD = 5
# Data file
LOG_FILE = '../logs.csv' We describe a function that collects a graph and sends it to the user: def send_pics(update, context, interval):

    try:
        # Get the user
        user = update.effective_user

        # Filter out bots (just in case)
        if user.is_bot:
            return

        # Check if the user is in the allowed list 
        username = str(user.username)
        if username not in ALLOWED_USERS:
            return
    except Exception as e:
        print(f'{repr(e)}')
        return

    # Read data from a file (file does not have a header)
    df = pd.read_csv(LOG_FILE, header=None, names=[
        'timestamp_string', 'max_val', 'max_support', 'med_val', 'med_support', 'block_number'
    ])
    # Convert a string to a date/time type
    df['timestamp'] = pd.to_datetime(df['timestamp_string'])

    # Determine the starting point from which the data will be reflected on the graph
    now = datetime.datetime.now()
    # Multiple intervals are available - week, day, hour
    if interval == 'week':
        one_x_ago = now - datetime.timedelta(weeks=1)
    elif interval == 'day':  # day
        one_x_ago = now - datetime.timedelta(days=1)
    else:
        one_x_ago = now - datetime.timedelta(hours=1)
    # Filter the dataframe
    df = df[df['timestamp'] >= one_x_ago]

    # Create columns with lags for the selected node and for the node with the highest block value
    cols_to_show = ['node_lag', 'best_node_lag']
    df['node_lag'] = df['block_number'] - df['med_val']
    df['best_node_lag'] = df['max_val'] - df['med_val']

    # Create a graph with seaborn
    plt.figure()
    sns.set(rc={'figure.figsize': (11, 4)})  # set figure size
    sns.lineplot(x='timestamp', y='value', hue='variable', data=df[['timestamp']+cols_to_show].melt('timestamp', var_name='variable', value_name='value'))
    # Add a "corridor" for easier comparison with the threshold
    plt.axhline(y=THRESHOLD, color='black', linestyle='--')
    plt.axhline(y=-THRESHOLD, color='black', linestyle='--')

    # Save the image to a buffer
    buf = io.BytesIO()
    plt.savefig(buf, format='png')
    buf.seek(0)

    # Send the image to the user
    context.bot.send_photo(chat_id=user.id, photo=buf)

    # "Close" the graph object
    plt.close() In this case, in the code, you can choose intervals for display: hour day week Their selection is carried out through the corresponding bot commands , , , which look like this in the code: /hour /day /week # Handler for the command to display the chart for the last hour
def hour(update, context):
    send_pics(update, context, 'hour')

# Handler for the command to display the chart for the last day
def day(update, context):
    send_pics(update, context, 'day')

# Handler for the command to display the chart for the last week
def week(update, context):
    send_pics(update, context, 'week') All that remains is to initialize the bot and "bind" the commands to their handlers: # Your Telegram bot token obtained through BotFather
token = "xxx"

# Create a bot instance
bot = telegram.Bot(token=token)
updater = Updater(token=token, use_context=True)
dispatcher = updater.dispatcher


# Connect command handlers to the bot's commands
dispatcher.add_handler(CommandHandler("hour", hour, filters=Filters.chat_type.private))
dispatcher.add_handler(CommandHandler("day", day, filters=Filters.chat_type.private))
dispatcher.add_handler(CommandHandler("week", week, filters=Filters.chat_type.private))

# Start the bot
updater.start_polling() In addition to this, for convenience, you can add real buttons for commands via BotFather. Finally the result looks like this: PS . I noticed that the labels under the time axis were overlapping, but this problem can be easily solved by adding a rotation of 15 degrees with the command "plt.xticks(rotation=15)", executed after plotting the graphs. That's all, if there are any questions, I will be glad to answer them. The source code of the project is available in the on GitHub. If this tutorial seemed useful to you, feel free to put a star on GitHub, I would appreciate it 🙂 repository

How to Create a Telegram Bot for Monitoring Your Service Uptime in Python (Part 3: Getting Graphs)

Hello everyone, in the , I started describing my experience of developing a Python Telegram bot for checking the operability and monitoring of my service located on a remote server. In a nutshell, when you're working on a pet project or even some work tasks, you might want to have all the current system status information at hand (I particularly like the ability to see everything and manage it through a Telegram bot) without spending a lot of time on development. previous article In the previous part, we looked at a way to get instant metrics on demand. In this part, we will be doing simple alerting, i.e., receiving a message in the bot when the system starts to fail. In the next third part, we will cover the case of collecting analytics and receiving online charts. As in the previous part, the example will be based on a real task, but I will be marking those places in the code that you can change to your logic so that the main part of the example can be reused. In this case, I need to get an alert in the form of a message in the Telegram bot in the event that the node I'm interested in loses network connection (or for some other reason), but its last synchronized block is lagging behind the network's last block. As before, we first need to set up the virtual environment: cd ~
virtualenv -p python3.8 up_env # creating an environment
source ~/up_env/bin/activate # activating the environment and install the necessary dependencies: pip install python-telegram-bot
pip install "python-telegram-bot[job-queue]" --pre
pip install --upgrade python-telegram-bot==13.6.0 # the code was written in the times before version 20, so here the version is explicitly specified

pip install numpy # needed for the median value function
pip install web3 # needed for requests to nodes (replace with what you need) The file does not undergo changes in this case and remains the same as in the previous part: functions.py import numpy as np
import multiprocessing

from web3 import Web3 #  add those libraries needed for your task

# Helper function that checks a single node
def get_last_block_once(rpc):
    try:
        w3 = Web3(Web3.HTTPProvider(rpc))
        block_number = w3.eth.block_number
        if isinstance(block_number, int):
            return block_number
        else:
            return None
    except Exception as e:
        print(f'{rpc} - {repr(e)}')
        return None


# Main function to check the status of the service that will be called
def check_service():
    # pre-prepared list of reference nodes
    # for any network, it can be found on the website https://chainlist.org/
    list_of_public_nodes = [
        'https://polygon.llamarpc.com',
        'https://polygon.rpc.blxrbdn.com',
        'https://polygon.blockpi.network/v1/rpc/public',
        'https://polygon-mainnet.public.blastapi.io',
        'https://rpc-mainnet.matic.quiknode.pro',
        'https://polygon-bor.publicnode.com',
        'https://poly-rpc.gateway.pokt.network',
        'https://rpc.ankr.com/polygon',
        'https://polygon-rpc.com'
    ]
    
    # parallel processing of requests to all nodes
    with multiprocessing.Pool(processes=len(list_of_public_nodes)) as pool:
        results = pool.map(get_last_block_once, list_of_public_nodes)
        last_blocks = [b for b in results if b is not None and isinstance(b, int)]

    # define the maximum and median value of the current block
    med_val = int(np.median(last_blocks))
    max_val = int(np.max(last_blocks))
    # determine the number of nodes with the maximum and median value
    med_support = np.sum([1 for x in last_blocks if x == med_val])
    max_support = np.sum([1 for x in last_blocks if x == max_val])

    return max_val, max_support, med_val, med_support Now let's look at the main bot file . Since in different tasks you may only need alerting or only request instant values, I don't build a bot that can do everything at once, but instead I divide this functionality into different small examples. In this case, the main bot file code will only include alerting, but you can combine everything you need into one bot. alert_bot.py So, we import libraries and functions from the file above and set the necessary constants: import telegram
from telegram.ext import Updater
from functions import get_last_block_once, check_service

The address of the node, the state of which I'm tracking (also a public node in this case)
OBJECT_OF_CHECKING = 'https://polygon-mainnet.chainstacklabs.com'

Threshold for highlighting critical lag
THRESHOLD = 5

Your Telegram account ID. The easiest way to find out is through the @chatIDrobot bot
USER_ID = 123456789 Next, we describe a function that will be called regularly by the timer: def check_for_alert(context):

    # Call of the main function to check the network state
    max_val, max_support, med_val, med_support = check_service()
    # Call of the function to check the state of the inspected node
    last_block = get_last_block_once(OBJECT_OF_CHECKING)

    # Forming a message to be sent to Telegram
    message = ""
    # Information about the state of nodes in the external network (median, maximum, and number of nodes)
    message += f"Public median block number {med_val} (on {med_support}) RPCs\n"
    message += f"Public maximum block number +{max_val - med_val} (on {max_support}) PRCs\n"

    # this variable will store the decision whether to send an alert
    # in case the node is lagging or didn't respond
    to_send = False

    # state check
    if last_block is not None:
        out_text = str(last_block - med_val) if last_block - med_val < 0 else '+' + str(last_block - med_val)
        # Comparison with the threshold
        if abs(last_block - med_val) > THRESHOLD:
            to_send = True
            message += f"The node block number shift ⚠️<b>{out_text}</b>⚠️"
        else:
            message += f"The node block number shift {out_text}"
    else: # Handling the exception if the node didn't respond
        to_send = True
        message += f"The node has ⚠️<b>not responded</b>⚠️"

    # triggering the alert and sending a message to the user
    if to_send:
        context.bot.send_message(chat_id=USER_ID, text=message, parse_mode="HTML") Next, you only have to write the part where the bot is initialized and a regular job checking the node's state is connected to it: # Your Telegram bot token obtained through BotFather
token = "xxx"

# creating a bot instance
bot = telegram.Bot(token=token)
updater = Updater(token=token, use_context=True)
dispatcher = updater.dispatcher
job_queue = updater.job_queue

# Here, the interval variable (in seconds) sets the frequency
# of launching the check - in this case, every 10 minutes
job_queue.run_repeating(check_for_alert, interval=10.0 * 60.0, first=0.0)

#bot start
updater.start_polling() Next, the code can be run on any VPS server through: source ~/up_env/bin/activate
python uptime_bot.py Having previously configured the systemd unit-file. As a result, the bot operation will look as follows - if an alert has been triggered, I receive a message with the problem information: In the following article, I will describe how to implement the remaining task: Retrieve graphs based on the request, showing how everything has been progressing over the past X hours/days. It will consist of two parts: a script to log events triggered by cron and a bot that collects graphs from the logs based on user requests. The source code of the project is available in the GitHub . If you found this tutorial helpful, feel free to star it on GitHub, I would appreciate it🙂 repository Also published . here

How to Create a Telegram Bot for Monitoring Your Service Uptime in Python (Part 2/3: Alerting)

Stories, News & Mentions about LLAMACORP

Llamacorp’s Stories on HackerNoon