K8s 节点组件部署：kubelet + kube-proxy + IPVS

Wed, 15 May 2024 00:00:00 +0800

kubelet + kube-proxy 是什么

kubelet 是每个 Node 上的"代理"，负责：

接收 apiserver 的 Pod spec
调用容器运行时（cri-dockerd）启动容器
上报节点状态、心跳、资源 metrics

kube-proxy 是每个 Node 上的"网络代理"，负责：

监听 apiserver 的 Service / Endpoints 变化
维护节点上 iptables 或 IPVS 规则
实现 ClusterIP、NodePort 的流量转发

适用版本：Kubernetes 1.28.5 / IPVS 部署位置：所有 master + worker 节点

1. kubelet 部署

1.1 systemd 单元

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


cat << "EOF" > /etc/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kubelet \
 --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig \
 --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
 --config=/etc/kubernetes/kubelet-conf.yml \
 --container-runtime-endpoint=unix:///run/cri-dockerd.sock \
 --node-labels=node.kubernetes.io/node=
Restart=always
RestartSec=5
StartLimitInterval=0

[Install]
WantedBy=multi-user.target
EOF

1.2 kubelet-conf.yml（关键）

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70


apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
authentication:
 anonymous:
 enabled: false
 webhook:
 cacheTTL: 2m0s
 enabled: true
 x509:
 clientCAFile: /etc/kubernetes/pki/ca.pem
authorization:
 mode: Webhook
 webhook:
 cacheAuthorizedTTL: 5m0s
 cacheUnauthorizedTTL: 30s
cgroupDriver: systemd  # 必须与 docker daemon.json 一致
cgroupsPerQOS: true
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
evictionHard:
 imagefs.available: 15%
 memory.available: 100Mi
 nodefs.available: 10%
 nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true # 强制要求 swap 已禁
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s

1.3 复制到所有节点

1
2
3
4
5
6


for NODE in master1 master2 master3 worker1 worker2 worker3 worker4 worker5 worker6 worker7 worker8 worker9; do
 scp /etc/kubernetes/kubelet-conf.yml $NODE:/etc/kubernetes/
 scp /etc/kubernetes/bootstrap-kubelet.kubeconfig $NODE:/etc/kubernetes/
 scp /etc/kubernetes/kube-proxy.kubeconfig $NODE:/etc/kubernetes/
 scp /etc/systemd/system/kubelet.service $NODE:/etc/systemd/system/
done

1.4 启动 kubelet

1
2
3
4
5


systemctl daemon-reload
systemctl enable --now kubelet.service
systemctl restart kubelet.service
systemctl status kubelet.service
journalctl -f -u kubelet

1.5 命令补全

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


cat << "EOF" >> /root/.bashrc
export KUBECONFIG=/etc/kubernetes/admin.kubeconfig
alias k=kubectl
source <(kubectl completion bash)
EOF

source /root/.bashrc

# 查看节点
kubectl get node

# 看容器运行时
kubectl describe node | grep Runtime

2. kube-proxy 部署

2.1 分发 kubeconfig

1
2
3


for NODE in master1 master2 master3; do
 scp /etc/kubernetes/kube-proxy.kubeconfig $NODE:/etc/kubernetes/
done

2.2 kube-proxy.service

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-proxy \
 --config=/etc/kubernetes/kube-proxy.yaml \
 --v=2
Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

2.3 kube-proxy.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
clientConnection:
 acceptContentTypes: ""
 burst: 10
 contentType: application/vnd.kubernetes.protobuf
 kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
 qps: 5
clusterCIDR: 172.218.0.0/12,fc00:2222::/112
configSyncPeriod: 15m0s
conntrack:
 max: null
 maxPerCore: 32768
 min: 131072
 tcpCloseWaitTimeout: 1h0m0s
 tcpEstablishedTimeout: 24h0m0s
enableProfiling: false
healthzBindAddress: 0.0.0.0:10256
hostnameOverride: ""
iptables:
 masqueradeAll: false
 masqueradeBit: 14
 minSyncPeriod: 0s
 syncPeriod: 30s
ipvs:
 masqueradeAll: true
 minSyncPeriod: 5s
 scheduler: "rr"
 syncPeriod: 30s
kind: KubeProxyConfiguration
metricsBindAddress: 127.0.0.1:10249
mode: "ipvs"
nodePortAddresses: null
oomScoreAdj: -999
portRange: ""
udpIdleTimeout: 250ms

2.4 启动

1
2
3
4
5


for NODE in master1 master2 master3 worker1 worker2 worker3 worker4 worker5 worker6 worker7 worker8 worker9; do
 ssh $NODE "systemctl daemon-reload"
 ssh $NODE "systemctl enable --now kube-proxy.service"
 ssh $NODE "systemctl restart kube-proxy.service"
done

2.5 验证 IPVS

1
2


ipvsadm -ln
# 预期看到 10.96.0.0/12 的 Service IP 列表

3. 节点 Ready 验证

1
2
3
4
5
6
7
8


# 等待 30 秒后
kubectl get node
# NAME STATUS ROLES AGE VERSION
# master1 Ready <none> 5m v1.28.5
# master2 Ready <none> 5m v1.28.5
# master3 Ready <none> 5m v1.28.5
# worker1 Ready <none> 3m v1.28.5
# ...

如果节点 NotReady：

1
2


kubectl describe node <node-name>
# 看 Conditions 里的 Ready 状态

常见原因：

cgroupDriver 不一致 → kubelet 起不来
CRI socket 路径错 → container-runtime-endpoint 检查
kubelet.conf 里 clusterDNS 写错 → CoreDNS IP 必须是 10.96.0.10

4. 排错速查

现象	原因	解决
`kubelet: failed to run Kubelet: validate service connection: validate CRI v1 runtime API`	cri-dockerd 没起	启动 cri-docker.service
`kubelet: cgroup driver "cgroupfs" is different from docker "systemd"`	docker cgroup driver 不一致	改 daemon.json + 重启 docker
`kubelet: node "master1" not found`	apiserver 没有这个 node 对象	`kubectl get csr` 看有没有待批准的证书
`kubelet: unauthorized`	bootstrap token 错	重新生成 bootstrap.secret.yaml
`failed to start CRI service: timeout`	cri-dockerd socket 不存在	检查 `/run/cri-dockerd.sock`
`kubelet.service: Main process exited, code=exited, status=255`	kubelet-conf.yml 写错	`kubelet --config=... --v=4` 看具体错

5. 小结

节点组件是 K8s 集群的"四肢"：

kubelet + cri-dockerd 缺一不可
cgroupDriver 必须 kubelet 与 docker 一致（都是 systemd）
IPVS 模式比 iptables 在大规模集群下性能更好
TLS Bootstrapping 用 token 代替每个 node 签证书

下一步：K8s 核心插件部署：Calico 网络 + CoreDNS + Metrics-server。

节点组件 on Liangweidong's blog

K8s 节点组件部署：kubelet + kube-proxy + IPVS

kubelet + kube-proxy 是什么

1. kubelet 部署

1.1 systemd 单元

1.2 kubelet-conf.yml（关键）

1.3 复制到所有节点

1.4 启动 kubelet

1.5 命令补全

2. kube-proxy 部署

2.1 分发 kubeconfig

2.2 kube-proxy.service

2.3 kube-proxy.yaml

2.4 启动

2.5 验证 IPVS

3. 节点 Ready 验证

4. 排错速查

5. 小结