生产环境离线部署大模型

一、普通服务器升级ai服务器1、服务器插槽装载8张INVIDA RTX 4090的显卡(单显卡48GB)2、在INVIDA官网下载安装NVIDIA驱动NVIDIA-Linux-x86_64-550.120.run//服务器上禁用开源驱动 nouveauecho blacklist nouveau /etc/modprobe.d/blacklist-nouveau.confecho options nouveau modeset0 /etc/modprobe.d/blacklist-nouveau.confupdate-initramfs -u//重启服务器reboot//安装invida的驱动./NVIDIA-Linux-x86_64-550.120.run --no-opengl-files --no-drm//安装完成后再重启服务器然后验证nvidia-smi出现8张显卡详情则说明安装成功3、在INVIDA官网下载安装cuda_12.1.0_530.30.02_linux.run//安装./cuda_12.1.0_530.30.02_linux.run//安装时一定取消勾选 Driver只装 Toolkit//配置环境echo export PATH/usr/local/cuda-12.1/bin:$PATH ~/.bashrcecho export LD_LIBRARY_PATH/usr/local/cuda-12.1/lib64:$LD_LIBRARY_PATH ~/.bashrcsource ~/.bashrc//验证ncvv -V4、下载安装其他环境//先安装pythontar -zxvf Python-3.10.14.tgz cd Python-3.10.14./configure --prefix/usr/local/python310make make installln -s /usr/local/python310/bin/python3 /usr/bin/pythonln -s /usr/local/python310/bin/pip3 /usr/bin/pip//安装下载好的离线包pip install --no-index --find-links. torch-2.4.0cu121-cp310-cp310-linux_x86_64.whlpip install --no-index --find-links. torchvision-0.19.0cu121-cp310-cp310-linux_x86_64.whl pip install --no-index --find-links. torchaudio-2.4.0cu121-cp310-cp310-linux_x86_64.whl pip install --no-index --find-links. vllm-0.7.2cu121-cp310-cp310-linux_x86_64.whlpip install --no-index --find-links. auto_gptq-0.7.1cu121-cp310-cp310-linux_x86_64.whl pip install --no-index --find-links. optimum-1.26.0-py3-none-any.whlpip install --no-index --find-links. transformers-4.48.0-py3-none-any.whlpip install --no-index --find-links. tokenizers-0.19.1-cp310-cp310-manylinux_2_17_x86_64.whlpip install --no-index --find-links. sentencepiece-0.2.0-cp310-cp310-manylinux_2_17_x86_64.whlpip install --no-index --find-links. protobuf-4.25.3-cp310-cp310-manylinux2014_x86_64.whl5、运行模型cd /data cat EOF start_235b.shCUDA_VISIBLE_DEVICES0,1,2,3 \python -m vllm.entrypoints.openai.api_server \--model /data/models/tclf90_Qwen3-235B-A22B-Thinking-2507-GPTQ-Int4-Int8Mix \--served-model-name qwen3-235b \--tensor-parallel-size 4 \--max-model-len 8192 \--gpu-memory-utilization 0.95 \--trust-remote-code \--host 0.0.0.0 \--port 8050EOF chmod x start_235b.sh nohup ./start_235b.sh run.log 21 6、直接测试curl http://127.0.0.1:8085/v1/chat/completions \-H Content-Type: application/json \-d { model: qwen3-235b, messages: [ {role: user, content: 你好我是一名AI工程师正在测试Qwen3-235B模型} ], temperature: 0.7, max_tokens: 1024 }7、在前端界面上实现智能对话//直接用开源的ChatGPT-Next-Web对接你的8050端口即可//或者开源的openwebui,然后调用大模型的接口# 下载WebUI离线包传到服务器后启动cd /data unzip chatgpt-next-web-v2.16.1-linux-amd64.zip//启动PORT3000 BASE_URLhttp://127.0.0.1:8050 API_KEYsk-123456 MODELqwen3-235b ./chatgpt-next-web//然后直接访问http://服务器IP:3000*知识点注释1、显卡与cpu显卡天生并行计算适合大模型cpu串行计算2、一台普通服务器如何变成ai服务器只需要加8张NVIDIA RTX 4090(每张显卡48GB)的显卡就立马变成一台顶级的ai服务器但是注意该服务器一定得解决一下问题:一般服务器只有2个左右插槽以及散热问题以及大功率电源