- 環境
已安裝nvidia 顯卡 驅動
操作系統:CentOS Linux release 7.9.2009 (Core)
內核版本:Linux 5.4.135-1.el7.elrepo.x86_64
顯卡 型號:rtx 6000
kvm版本:
virsh version
Compiled against library: libvirt 4.5.0
Using library: libvirt 4.5.0
Using API : QEMU 4.5.0
Running hypervisor: QEMU 1.5.3
- 解綁GPU 設備
- 配置GRUB_CMDLINE_LINUX(實測，有效)，查詢要解綁的設備，記錄設備id號
lspci -nn|grep NVIDIA |grep VGA
或者 lspci -nn|grep NVIDIA
2.修改 /etc/default/grub文件的GRUB_CMDLINE_LINUX
vi /etc/default/grub
以下是內容，新增加 GRUB_CMDLINE_LINUX行: intel_iommu=on pci-stub.ids=10de:1e30"
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="intel_iommu=on crashkernel=auto spectre_v2=retpoline rhgb quiet rdblacklist=nouveau pci-stub.ids=10de:1e30"
GRUB_DISABLE_RECOVERY="true"
3.更新grub2 conf，執行如下命令
grub2-mkconfig -o /boot/grub2/grub.cfg
4.重啓系統
reboot
- 驗證並查看nvidia 顯卡
1.執行nvidia-smi ，看不到顯卡，證明顯卡已經被解綁
# nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
2.驗證內核支持iommu
cat /proc/cmdline | grep iommu
結果輸出: intel_iommu=on
3.驗證VT-D是否啓用
dmesg |grep -e DMAR -e IOMMU
- 安裝win10虛擬機
qemu-img create -f qcow2 /home/kvm-machines/shaoyan.qcow2 200G
virt-install --name shaoyan --ram 4096 --vcpus=8 --os-type=windows --accelerate --cdrom=/home/kvm_images/win10.iso --disk path=/home/kvm-machines/shaoyan.qcow2,format=qcow2,bus=ide --bridge=br0 --autostart --vnc --vncport=5910 --vnclisten=0.0.0.0
- 宿主機安裝軟件包
yum -y groups install 'GNOME Desktop' virt-manager virt-viewer dejavu-lgc-sans-fonts
- 顯卡加載vfio驅動
1.顯卡加載vfio驅動
modprobe vfio
modprobe vfio-pci
2.從服務器卸載1張顯卡的4個設備
virsh nodedev-detach pci_0000_b1_00_0
virsh nodedev-detach pci_0000_b1_00_1
virsh nodedev-detach pci_0000_b1_00_2
virsh nodedev-detach pci_0000_b1_00_3
3.驗證顯卡設備是否加載vfio-pci驅動,加載成功
# lspci -vv -s b1:00.0 | grep driver
Kernel driver in use: vfio-pci
[root@rtx_6000 10.147.100.107 ~ ]
# lspci -vv -s b1:00.1 | grep driver
Kernel driver in use: vfio-pci
[root@rtx_6000 10.147.100.107 ~ ]
# lspci -vv -s b1:00.2 | grep driver
Kernel driver in use: vfio-pci
[root@rtx_6000 10.147.100.107 ~ ]
# lspci -vv -s b1:00.3 | grep driver
Kernel driver in use: vfio-pci
- win10虛擬機添加NVDIA顯卡
1.通過遠控卡登錄服務器透傳顯卡，執行startx 調出可視化頁面
startx
2.添加顯卡硬件
3.查看顯卡
- 顯卡驅動安裝
1.要安裝顯卡驅動，需要修改虛擬機win10的配置文件 否則顯卡驅動安裝過程中自檢會報錯ERROR CODE 43，驅動不允許安裝在虛擬機上。
#編輯VM XML
virsh edit shaoyan
#修改第一行
<domain type='kvm'>
#爲
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
#找到<features>節點，添加內容
<features>
...
<kvm><hidden state='on'/></kvm>
...
</features>
#找到</device>節點，在device後添加內容
<qemu:commandline><qemu:arg value='-cpu'/><qemu:arg value='host,hv_time,kvm=off,hv_vendor_id=null'/>
</qemu:commandline>
#重啓VM
virsh destroy shaoyan
virsh start shaoyan
2.附錄完整xml 文件如下，如果是拷貝至其他宿主機配置文件的話，只需要保留xml 文件中的第一組hostdev即可，否則會報錯
# cat /etc/libvirt/qemu/shaoyan.xml
<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
virsh edit shaoyan
or other application using the libvirt API.
-->
<domain type='kvm'>
<name>shaoyan</name>
<uuid>e06c6c3b-0a57-43e8-9d6f-02fd1e2aad83</uuid>
<title>gpu</title>
<description>gpu</description>
<memory unit='KiB'>16777216</memory>
<currentMemory unit='KiB'>16777216</currentMemory>
<vcpu placement='static'>16</vcpu>
<os>
<type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<hyperv>
<relaxed state='on'/>
<vapic state='on'/>
<spinlocks state='on' retries='8191'/>
</hyperv>
</features>
<cpu mode='host-passthrough' check='none'/>
<clock offset='localtime'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
<timer name='hypervclock' present='yes'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/qemu-kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/data/kvm/shaoyan.qcow2'/>
<target dev='hda' bus='ide'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<controller type='usb' index='0' model='ich9-ehci1'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x7'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci1'>
<master startport='0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci2'>
<master startport='2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci3'>
<master startport='4'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'/>
<controller type='ide' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<interface type='bridge'>
<mac address='52:54:00:48:b2:5a'/>
<source bridge='br0'/>
<model type='e1000'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<serial type='pty'>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<input type='tablet' bus='usb'>
<address type='usb' bus='0' port='1'/>
</input>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' port='5910' autoport='no' listen='0.0.0.0'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<video>
<model type='vga' vram='16384' heads='1' primary='yes'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0xb1' slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0xb1' slot='0x00' function='0x1'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0xb1' slot='0x00' function='0x2'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0xb1' slot='0x00' function='0x3'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</hostdev>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</memballoon>
</devices>
</domain>
3.登錄win10 虛擬機配置網絡
4.安裝驅動，選擇具體型號進行下載並安裝驅動
https://www.nvidia.cn/Download/index.aspx?lang=cn
5.查看顯卡設備
6.xmanage 軟件下載
鏈接：https://pan.baidu.com/s/1WVN2fqThOpozecoeJtktiQ
提取碼：0035
7.安裝xmanage 並啓動 xmanager.exe
8.服務器修改 /etc/ssh/sshd_config,修改兩項
X11Forwarding yes
X11UseLocalhost no
9.啓動桌面,10.33.0.214 地址爲本地 openvpn 分配的私網地址
export DISPLAY=10.33.0.214:0.0
virt-manager
- 至此，win10通過kvm虛擬化透傳GPU 顯卡完成！