PVE 安装教程

购入一台戴尔 R630, 安装 PVE

背景

购入一台二手戴尔 R630

• CPU: Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz X2
• Memory: 256GB ECC DDR4 2400 MHz
• Disk: SSD 1.8TB X2 (Raid0), HDD 558GB X6 (Raid10)

从配置上来看并不稀奇，双路 CPU 共计 40 核心 80 线程，插了 8 条 32GB 内存，希望通过它深入的学习 k8s。

安装 PVE

在 R630 上安装 PVE 和传统的 PC 主机有所区别。对于 PC 主机而言，可以直接通过制作 PVE U 盘启动器安装；R630 安装 PVE 的方式可以参考这篇帖子：通过 idrac 安装 PVE

在安装过程中，我将 PVE 安装在了虚拟磁盘 SSD_RAID0_1.8T 上，该虚拟磁盘由 H730p mini 阵列卡将两块三星 870EVO 固态硬盘使用 Raid0 模式组合而成。

配置订阅源并删除弹窗

# 将此文件的中的所有内容注释掉
nano /etc/apt/sources.list.d/pve-enterprise.list

# 下载中科大的GPG KEY
wget https://mirrors.ustc.edu.cn/proxmox/debian/proxmox-release-bookworm.gpg -O /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg

# 使用Proxmox非企业版源
echo "deb https://mirrors.ustc.edu.cn/proxmox/debian bookworm pve-no-subscription" > /etc/apt/sources.list.d/pve-no-subscription.list

# 将Debian官方源替换为中科大源
cp /etc/apt/sources.list /etc/apt/sources.list-bak
sed -i 's|^deb http://ftp.debian.org|deb https://mirrors.ustc.edu.cn|g' /etc/apt/sources.list
sed -i 's|^deb http://security.debian.org|deb https://mirrors.ustc.edu.cn/debian-security|g' /etc/apt/sources.list

# 替换Ceph源
echo "deb https://mirrors.ustc.edu.cn/proxmox/debian/ceph-quincy bookworm no-subscription" > /etc/apt/sources.list.d/ceph.list

# 替换CT镜像下载源
sed -i 's|http://download.proxmox.com|https://mirrors.ustc.edu.cn/proxmox|g' /usr/share/perl5/PVE/APLInfo.pm

#删除订阅弹窗
sed -Ezi.bak "s/(Ext.Msg.show\(\{\s+title: gettext\('No valid sub)/void\(\{ \/\/\1/g" /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js && systemctl restart pveproxy.service
# 执行完成后，浏览器Ctrl+F5强制刷新缓存

其它

使用 fio 查看磁盘 IOPS，大概有 100K，还算可以吧。

root@pve:~# fio -direct=1 -iodepth=32 -rw=randread -ioengine=io_uring -bs=4k -size=10G -numjobs=4 -runtime=10000 -group_reporting -filename=iotest -name=Rand_Read_Testing
Rand_Read_Testing: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=32
...
fio-3.33
Starting 4 processes
Rand_Read_Testing: Laying out IO file (1 file / 10240MiB)
Jobs: 3 (f=3): [r(3),_(1)][100.0%][r=434MiB/s][r=111k IOPS][eta 00m:00s]
Rand_Read_Testing: (groupid=0, jobs=4): err= 0: pid=21778: Sun Jun  9 20:16:25 2024
  read: IOPS=122k, BW=476MiB/s (499MB/s)(40.0GiB/86110msec)
    slat (usec): min=9, max=555, avg=25.51, stdev=12.04
    clat (usec): min=18, max=17256, avg=1015.18, stdev=424.11
     lat (usec): min=44, max=17283, avg=1040.69, stdev=423.77
    clat percentiles (usec):
     |  1.00th=[  816],  5.00th=[  889], 10.00th=[  914], 20.00th=[  947],
     | 30.00th=[  971], 40.00th=[  988], 50.00th=[ 1004], 60.00th=[ 1012],
     | 70.00th=[ 1029], 80.00th=[ 1057], 90.00th=[ 1090], 95.00th=[ 1123],
     | 99.00th=[ 1188], 99.50th=[ 1254], 99.90th=[12649], 99.95th=[13173],
     | 99.99th=[13435]
   bw (  KiB/s): min=44096, max=590488, per=100.00%, avg=489534.41, stdev=14286.68, samples=683
   iops        : min=11024, max=147622, avg=122383.62, stdev=3571.67, samples=683
  lat (usec)   : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.02%
  lat (usec)   : 750=0.29%, 1000=49.73%
  lat (msec)   : 2=49.79%, 4=0.03%, 10=0.01%, 20=0.12%
  cpu          : usr=13.60%, sys=77.61%, ctx=795992, majf=0, minf=655
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=10485760,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=476MiB/s (499MB/s), 476MiB/s-476MiB/s (499MB/s-499MB/s), io=40.0GiB (42.9GB), run=86110-86110msec

Disk stats (read/write):
    dm-1: ios=10485691/176, merge=0/0, ticks=9673094/163, in_queue=9673257, util=99.59%, aggrios=10485789/110, aggrmerge=3/66, aggrticks=9624816/108, aggrin_queue=9624925, aggrutil=99.58%
  sdb: ios=10485789/110, merge=3/66, ticks=9624816/108, in_queue=9624925, util=99.58%