Munguva pfupi yapfuura, pane kudiwa kweyakavanzika chatbot sevhisi seimwe nzira yakakwana kune OpenAI's ChatGPT. Saka, ini ndinosarudza kuita imwe kumba uye kuita kuti iwanikwe kune wese wemba yangu padivi nenetiweki printer yangu uye . NAS (OpenMediaVault) Kare, ndaikurudzira vanhu kushandisa Llama akatevedzana echiRungu mabasa uye Qwen akatevedzana echiChinese mabasa. Pakanga pasina yakavhurika-sosi modhi yakasimba zvakakwana mumitauro yakawanda tichienzanisa neyevaridzi (GPT/Claude). Zvisinei, sezvatinoziva tose—zvinhu zvakachinja munguva pfupi yapfuura. Ndanga ndichishandisa DeepSeek-V2 apo neapo pese pandakaneta neQwen2.5 uye ndanga ndichidzokera kumashure neDeepSeek V2.5 uye V3 nekuda kwekushaikwa kwehardware. Asi DeepSeek haina kundisiya pasi, R1 inoita zvinokatyamadza uye inopa diki se1.5B! Izvi zvinoreva kuti tinogona kuimhanyisa kunyangwe paCPU neruzivo rwakakura rwemushandisi. Sezvo vanhu vazhinji vaine maGPU ekutamba, kumhanya haisi nyaya. Kugadzira maLLM emunharaunda maitiro akarodha magwaro nemifananidzo mukana wakakura sezvo OpenAI ichimisa kushandiswa kwemaakaundi emahara. Kunyangwe iri nyore kwazvo neiyo yepamutemo-mutsara wekuraira: Kuisa Vhura WebUI neBundled Ollama Tsigiro docker run -d -p 3000:8080 --gpus=all -v ollama:/root/.ollama -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:ollama Asi kuwana RAG (Webhu yekutsvaga) kushanda hakusi nyore kune vanhu vazhinji, saka ndinoda kuwana imwe kunze kwebhokisi mhinduro. Sezvandakataura mune , chiteshi chengarava chiyedzo chikuru chekuyedza akasiyana LLM stack. Asi hazvina kunaka chete kune izvo, zvakare iri-mu-imwe mhinduro yekuzvigashira wega maLLM emunharaunda neRAG inoshanda kunze kwebhokisi. Saka, ngatitangei kuishandisa kubva kutanga uye tinzwe wakasununguka kusvetuka nhanho sezvo vanhu vazhinji vasingatangi kubva kuOS yekumisikidza. yangu yekupedzisira positi Kugadzirira Sistimu (Inosarudzika) Sezvakangoita , enda kuburikidza nekuisa maitiro uchishandisa sepakutanga debian-11.6.0-amd64-netinst.iso Wedzera kune sudoer wobva watangazve usermod -aG sudo username (Sarudzo) Wedzera imwe shanduko fallocate -l 64G /home/swapfile chmod 600 /home/swapfile mkswap /home/swapfile swapon /home/swapfile uye ita kuti swapfile irambe iripo nano /etc/fstab UUID=xxxxx-xxx swap swap defaults,pri=100 0 0 /home/swapfile swap swap defaults,pri=10 0 0 Tarisa ne kana swapon --show free -h Dzima mutyairi weNouveau bash -c "echo blacklist nouveau > /etc/modprobe.d/blacklist-nvidia-nouveau.conf" bash -c "echo options nouveau modeset=0 >> /etc/modprobe.d/blacklist-nvidia-nouveau.conf" update-initramfs -u update-grub reboot Isa zvinoenderana apt install linux-headers-`uname -r` build-essential libglu1-mesa-dev libx11-dev libxi-dev libxmu-dev gcc software-properties-common sudo git python3 python3-venv pip libgl1 git-lfs -y (Sarudzo) Ita uninstall kana zvichidikanwa apt-get purge nvidia* apt remove nvidia* apt-get purge cuda* apt remove cuda* rm /etc/apt/sources.list.d/cuda* apt-get autoremove && apt-get autoclean rm -rf /usr/local/cuda* Isa cuda-tokit uye cuda wget https://developer.download.nvidia.com/compute/cuda/12.4.1/local_installers/cuda-repo-debian11-12-4-local_12.4.1-550.54.15-1_amd64.deb sudo dpkg -i cuda-repo-debian11-12-4-local_12.4.1-550.54.15-1_amd64.debsudo cp /var/cuda-repo-debian11-12-4-local/cuda-*-keyring.gpg /usr/share/keyrings/ sudo add-apt-repository contrib sudo apt-get update sudo apt-get -y install cuda-toolkit-12-4 sudo apt install libxnvctrl0=550.54.15-1 sudo apt-get install -y cuda-drivers sezvo chiteshi chiri docker-yakavakirwa Isa iyo NVIDIA Container Toolkit curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \ && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \ sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \ sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list Zvadaro uye sudo apt-get update sudo apt-get install -y nvidia-container-toolkit Ita cuda post-install chiito nano ~/.bashrc export PATH=/usr/local/cuda-12.4/bin${PATH:+:${PATH}} export LD_LIBRARY_PATH=/usr/local/cuda-12.4/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}} Zvadaro , kana kushandisa shanduko sudo update-initramfs -u ldconfig source ~/.bashrc mushure mekutangazve, simbisa ne uye nvidia-smi nvcc --version Isa Miniconda (Sarudzo, kwete yechiteshi) wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh && sudo chmod +x Miniconda3-latest-Linux-x86_64.sh && bash Miniconda3-latest-Linux-x86_64.sh Docker & Harbour Isa docker # Add Docker's official GPG key: sudo apt-get update sudo apt-get install ca-certificates curl sudo install -m 0755 -d /etc/apt/keyrings sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc sudo chmod a+r /etc/apt/keyrings/docker.asc # Add the repository to Apt sources: echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \ $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \ sudo tee /etc/apt/sources.list.d/docker.list > /dev/null sudo apt-get update sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin Ita ye docker pasina sudo post-install sudo groupadd docker sudo usermod -aG docker $USER newgrp docker docker run hello-world Manual install Harbor git clone https://github.com/av/harbor.git && cd harbor ./harbor.sh ln Simbisa harbor --version Wedzera neRAG tsigiro kune defaults harbor defaults add searxng Shandisa kutarisa, ikozvino kune matatu masevhisi anoshanda: , , harbor defaults list ollama webui searxng Mhanya kuunza masevhisi aya mudocker harbor up Shandisa se , uye kuti uone matanda anorema harbor ps docker ps harbor logs Ikozvino iyo yakavhurika-webui yekumberi iri kushanda uye inogona kuwanikwa kubva kana vatengi kubva kuLAN ine server's IP kero. 0.0.0.0:33801 http://localhost:33801 Tarisa kushandiswa kweVRAM watch -n 0.3 nvidia-smi Monitor log kana harbor up ollama --tail harbor logs Yese mirairo yeollama inoshandiswa senge harbor ollama list Yave nguva yekuwana kubva kune mamwe maturusi (desktop/mobile) kunyoresa admin account uye kurodha modhi izvozvi. Kushandisa Local LLM Mushure mekupinda neakaundi ye admin, tinya kumusoro kurudyi avatar icon, vhura ipapo , kana kungo pinda kuburikidza `http://ip:33801/admin/settings. Admin Panel Settings Dzvanya , uye kumusoro kurudyi tinya inoita sebhatani rekurodha. Models Manage Models Isa kana mubhokisi rezvinyorwa pazasi uye tinya bhatani rekurodha padivi rekurudyi. deepseek-r1 chero imwe modhi Pull a model from Ollama.com Mushure mekudhawunirodha modhi, inogona kuzorodzwa uye iyo ichangobva kudhawunirodha modhi ichashandiswa pasi peiyo inodonha menyu paNew (yekumba) peji. New Chat Ikozvino, haisi kungomhanyisa chatbot imwe nzira yeChatGPT, asiwo inoshanda zvizere API imwe nzira yeOpenAI API, pamwe neyakavanzika yekutsvaga injini imwe nzira kuGoogle! webui inowanikwa mukati meLAN kuburikidza ne: http://ip:33801 ollama inowanikwa mukati meLAN kuburikidza ne: http://ip:33821 searching inowanikwa mukati meLAN kuburikidza ne: http://ip:33811 Fonera nechero application ine LLM API yekubatanidza: Ollama API http://ip:33821/api/ps http://ip:33821/v1/models http://ip:33821/api/generate http://ip:33821/v1/chat/completionsb
