How to install Gemma 4, QWEN 2.5-72B and QWEN 2.5-VL-72B, QWEN 3.5 locally on Dual RTX 3090
Vibe code is everywhere, and this time of writing QWEN 2.5 with 72B parameter, here is how to do it in Dual RTX 3090.
Before begin to install, please set your connection as stable as you can, or you can set the connection to use bigger MTU
ENVIRONMENT SETUP
Set the MTU of the WIFI, so it will have stable connection
ip link show # showing all the connections in the machine, if the MTU 1000, just make it bigger
sudo ip link set dev eth0 mtu 1400 # change the eth0 with the connection you use to download
export PIP_NO_IPV6=1 # to use ipv4 only instead of ipv6, then run below commands
Make sure the you use nvidia driver version 555
sudo apt install nvidia-driver-555And then create virtual environment in python
sudo apt update && sudo apt install -y git wget python3-venv python3-pip
python3 -m venv kimi_env && source kimi_env/bin/activate
pip install vllm --extra-index-url https://pypi.org/simple
or
# below is faster and get nightly build
uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate
uv pip install -U vllm --torch-backend=auto --extra-index-url https://wheels.vllm.ai/nightly # add variant subdirectory here if neededRUNNING THE MODEL
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
export NCCL_IB_DISABLE=0
export VLLM_ATTENTION_BACKEND=FLASH_ATTN
export VLLM_USE_TORCH_COMPILE=1
vllm serve \
--model Qwen/Qwen2.5-72B-Instruct-AWQ \
--tensor-parallel-size 2 \
--max-model-len 16384 \
--gpu-memory-utilization 0.96 \
--kv-cache-dtype fp8 \
--enforce-eager \
--max-num-batched-tokens 4096 \
--trust-remote-codeTEST THE LOCAL API
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-72B-Instruct-AWQ",
"messages": [{"role": "user", "content": "Write a fast Python function to find the nth Fibonacci number."}],
"max_tokens": 100
}'if you want to run QWEN 2.5-VL-72B, you can run below command
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export VLLM_IMAGE_MAX_PIXELS=602112
vllm serve \
--model Qwen/Qwen2.5-VL-72B-Instruct-AWQ \
--tensor-parallel-size 2 \
--max-model-len 8192 \
--kv-cache-dtype fp8 \
--limit-mm-per-prompt '{"image": 5}' \
--gpu-memory-utilization 0.95 \
--enforce-eager \
--trust-remote-codeIf you want to run QWEN 3.5-27B-Claude-4.6-Opus-Reasoning-Distilled-v2-AWQ, you can use this command.
export VLLM_IMAGE_MAX_PIXELS=602112
vllm serve QuantTrio/Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled-v2-AWQ --tensor-parallel-size 2 --max-model-len 65535 --tokenizer Qwen/Qwen3.5-27B --kv-cache-dtype fp8 --gpu-memory-utiliza
tion 0.90 --enforce-eager --trust-remote-codeif you want to run QWEN 3.5-35B-A3B, you can run below command, but seems to slow in my box ‼️
export VLLM_IMAGE_MAX_PIXELS=602112
vllm serve cyankiwi/Qwen3.5-35B-A3B-AWQ-4bit \
--tensor-parallel-size 2 \
--max-model-len 65536 \
--kv-cache-dtype fp8 \
--mm-encoder-tp-mode data \
--gpu-memory-utilization 0.95 \
--trust-remote-code
--enforce-eagerGEMMA 4 has just launched, and this is the best LLM open source model running in my dual RTX 3090 👑👑
export HF_TOKEN=your_token_here
export NCCL_P2P_DISABLE=1
export NCCL_SHM_DISABLE=1
export VLLM_USE_V1=0
uv run vllm serve cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit \
--tensor-parallel-size 2 \
--max-model-len 65536 \
--tokenizer google/gemma-4-26B-A4B-it \
--kv-cache-dtype auto \
--gpu-memory-utilization 0.90 \
--trust-remote-code \
--enforce-eager
#or if you want to use with claude code
uv run vllm serve cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit \
--tensor-parallel-size 2 \
--max-model-len 128000 \
--tokenizer google/gemma-4-26B-A4B-it \
--kv-cache-dtype auto \
--gpu-memory-utilization 0.95 \
--trust-remote-code \
--enforce-eager \
--enable-auto-tool-choice \
--tool-call-parser gemma4 \
--reasoning-parser gemma4
ADD UI INTERFACE BY RUNNING OPEN WEBUI
Run open webui docker
docker run -d -p 3000:8080 \
--add-host=host.docker.internal:host-gateway \
-v open-webui:/app/backend/data \
--name open-webui \
--restart always \
ghcr.io/open-webui/open-webui:mainCONFIGURE THE OPEN WEB UI
Run searxng docker to handle the web search, so the LLM can connect to the real-time events.
docker run -d -p 8888:8080 \
--name searxng \
--restart always \
-v searxng_data:/etc/searxng searxng/searxng:latestRun another docker image of Docling, to convert the PDF into image, since QWEN 2.5-VL only support reading document in image format, so we need to convert the PDF to image first.
docker run -d -p 5001:5001 \
--name docling-serve \
--restart always \
-e DOCLING_SERVE_ENABLE_UI=true \
-e DOCLING_SERVE_ENABLE_REMOTE_SERVICES=true \
-e DOCLING_SERVE_API_KEY=boringtaskai quay.io/docling-project/docling-serve:latest