K8s 集群升级与卸载：版本升级 + finalizers 清理

Fri, 15 Nov 2024 00:00:00 +0800

升级 vs 卸载

K8s 集群"生命周期"两个最复杂的操作：

升级：小版本（1.28.5 → 1.28.8）可以"无脑"替换二进制；大版本（1.28 → 1.29）要看组件参数变化
卸载：彻底清掉所有 master / worker 上的 K8s 文件、systemd 单元、容器运行时、证书

适用版本：升级 1.28.5 → 1.28.8 / 卸载 K8s 1.28.5

1. 升级策略

1.1 大版本 vs 小版本

小版本升级（patch 1.28.x → 1.28.y）：只换二进制 + 重启组件
次版本升级（1.28 → 1.29）：要关注 deprecated API（如 policy/v1beta1）、CRD schema 变化、组件 flag 变化
大版本升级（1.x → 2.x）：官方不支持，必须重装

1.2 升级顺序

1

master1 → master2 → master3 → worker1 → worker2 → ... → workerN

etcd 先升，K8s 控制面组件（apiserver → controller-manager → scheduler）后升，最后 worker。

1.3 当前环境

组件	当前	目标
etcd	v3.5.11	3.5.13
kubernetes-server	1.28.5	1.28.8
coredns	1.29.0	1.29.0（不动）

2. 升级 etcd

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


# 备份
mkdir -p /data/etcd-bak
etcdctl snapshot save /data/etcd-bak/etcd-snapshot-$(date +%Y%m%d).db

# 停 etcd
systemctl stop etcd.service
mv /usr/local/bin/etcd* /data/etcd-bak/

# 装新版本
cd /data/softs
tar -xf etcd-v3.5.13-linux-amd64.tar.gz
mv etcd-v3.5.13-linux-amd64/etcd /usr/local/bin/
mv etcd-v3.5.13-linux-amd64/etcdctl /usr/local/bin/
mv etcd-v3.5.13-linux-amd64/etcdutl /usr/local/bin/

# 同步到其他 master
for NODE in master2 master3; do
 ssh $NODE "mv /usr/local/bin/etcd* /data/etcd-bak/"
 scp /usr/local/bin/etcd* $NODE:/usr/local/bin/
done

# 启动
systemctl start etcd.service
etcdctl version
# etcdctl version: 3.5.13

3. 升级 K8s 控制面

3.1 升级 master1

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


# 备份
mkdir -p /usr/local/bin/bak
mv /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} /usr/local/bin/bak

# 装新版本
cd /data/softs
tar -xf kubernetes-server-linux-amd64.tar.gz \
 --strip-components=3 \
 -C /usr/local/bin \
 kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}

# 重启所有 master 组件
systemctl restart kube-apiserver.service
systemctl restart kube-controller-manager.service
systemctl restart kube-scheduler.service
systemctl restart kubelet.service
systemctl restart kube-proxy.service

# 验证
kubectl get node
# master1 应该还是 Ready，但 VERSION 还是 1.28.5（只有 etcd 那台没升时）

3.2 升级 master2 + master3

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


for NODE in master2 master3; do
 echo "=== $NODE ==="
 ssh root@$NODE "mkdir -p /usr/local/bin/bak && \
 mv /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} /usr/local/bin/bak"
 scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} \
 $NODE:/usr/local/bin/
 ssh root@$NODE "systemctl restart kube-apiserver.service && \
 systemctl restart kube-controller-manager.service && \
 systemctl restart kube-scheduler.service && \
 systemctl restart kubelet.service && \
 systemctl restart kube-proxy.service"
done

3.3 升级 worker

1
2
3
4
5
6
7
8


for NODE in worker1 worker2 worker3 worker4 worker5 worker6 worker7 worker8 worker9; do
 echo "=== $NODE ==="
 ssh root@$NODE "mkdir -p /usr/local/bin/bak && \
 mv /usr/local/bin/kube{let,-proxy} /usr/local/bin/bak"
 scp /usr/local/bin/kube{let,-proxy} $NODE:/usr/local/bin/
 ssh root@$NODE "systemctl restart kubelet.service && \
 systemctl restart kube-proxy.service"
done

3.4 验证

1
2


kubectl get node
# 全部 Ready，VERSION 显示 1.28.8

4. 升级时的节点隔离

升级 master 时要先把 master 标记不可调度（避免新 Pod 调度到正在重启的 master）：

1
2
3
4
5
6
7
8


# 隔离
kubectl cordon master1
kubectl drain master1 --delete-local-data --ignore-daemonsets --force
systemctl restart kube-apiserver.service
# ...

# 恢复
kubectl uncordon master1

drain 驱逐所有非 DaemonSet Pod 到其他节点。--delete-emptydir-data 允许删除 emptyDir 类型数据（一般安全）。

5. 卸载 K8s 集群

5.1 停服务顺序

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# 1. 停所有 K8s 服务
systemctl status kube-apiserver.service
systemctl status kube-controller-manager.service
systemctl status kube-scheduler.service
systemctl status kubelet.service
systemctl status kube-proxy.service

# 停
systemctl stop kube-apiserver.service
systemctl disable kube-apiserver.service
# ... 同上其他

5.2 删 systemd 单元

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


rm -rf /etc/systemd/system/kube-apiserver.service
rm -rf /etc/systemd/system/multi-user.target.wants/kube-apiserver.service
rm -rf /etc/systemd/system/kube-controller-manager.service
rm -rf /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service
rm -rf /etc/systemd/system/kube-scheduler.service
rm -rf /etc/systemd/system/multi-user.target.wants/kube-scheduler.service
rm -rf /etc/systemd/system/kubelet.service
rm -rf /etc/systemd/system/multi-user.target.wants/kubelet.service
rm -rf /etc/systemd/system/kube-proxy.service
rm -rf /etc/systemd/system/multi-user.target.wants/kube-proxy.service
rm -rf ~/bootstrap
systemctl daemon-reload

5.3 删目录

1
2
3
4
5
6
7


rm -rf /etc/kubernetes
rm -rf /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}
rm -rf /var/lib/kubelet
rm -rf /var/log/kubernetes
rm -rf /etc/cni/net.d
rm -rf /root/.kube
rm -rf ~/bootstrap/bootstrap.secret.yaml

5.4 清环境变量

1
2
3
4
5
6


# 临时
unset KUBECONFIG

# 永久
sudo sed -i '/KUBECONFIG/d' /etc/profile ~/.bashrc ~/.profile
source ~/.bashrc

5.5 停 etcd

1
2
3
4
5
6
7
8


systemctl status etcd.service
systemctl stop etcd.service
systemctl disable etcd.service
rm -rf /etc/systemd/system/etcd.service
rm -rf /usr/local/bin/etcd*
rm -rf /etc/etcd
rm -rf /var/lib/etcd
systemctl daemon-reload

5.6 停容器运行时

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


# 停所有 docker 相关
systemctl stop docker.service
systemctl disable docker.service
systemctl stop cri-docker.socket
systemctl disable cri-docker.socket
systemctl stop cri-docker.service
systemctl disable cri-docker.service
systemctl stop containerd.service
systemctl disable containerd.service

# 删 systemd 单元
rm -rf /etc/systemd/system/{docker.service,docker.socket,cri-docker.service,cri-docker.socket,containerd.service}
rm -rf /etc/docker
rm -rf /usr/local/bin/{containerd*,docker*,runc,cri-dockerd,ctr}
rm -rf /var/lib/docker
rm -rf /var/lib/containerd
rm -rf /run/cri-dockerd.sock
rm -rf /var/run/docker.sock

systemctl daemon-reload

5.7 其他清理

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# 网络（IPIP 模式 / CNI）
modprobe -r ipip 2>/dev/null
ip link delete tunl0 2>/dev/null
rm -rf /opt/cni/bin
rm -rf /var/lib/cni
rm -rf /var/lib/calico 2>/dev/null

# bootstrap 目录
rm -rf ~/bootstrap

# 历史镜像
docker system prune -af 2>/dev/null

5.8 验证

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# 进程
ps -ef | grep -E "kube|etcd|docker|containerd" | grep -v grep
# 应当空

# 端口
ss -tlnp | grep -E "6443|2379|2380|10250|10251|10252|10255"
# 应当空

# 残留目录
ls /usr/local/bin | grep -E "kube|etcd|containerd|docker"
# 应当空

6. namespace 卡 Terminating 终极清理

卸载过程中如果 namespace 一直卡 Terminating（典型场景：Rook-Ceph、cert-manager）：

1
2
3
4
5


# 通用方法
ns=rook-ceph
kubectl get namespace $ns -o json | jq '.spec = {"finalizers":[]}' > temp.json
curl -k -H "Content-Type: application/json" -X PUT \
 --data-binary @temp.json 127.0.0.1:8001/api/v1/namespaces/$ns/finalize

或：

1

kubectl patch ns rook-ceph --type=merge -p '{"metadata":{"finalizers":null}}'

但要先启动 kubectl proxy：

1

kubectl proxy &

6.1 PVC 卡 Terminating

1
2
3
4


kubectl patch pvc jenkins-agent-pvc -n kube-ops --type=merge -p '{"metadata":{"finalizers":null}}'

kubectl patch pv pvc-c87554c7-6b20-42ed-8cdd-8be1d630a744 \
 -p '{"metadata":{"finalizers":null}}' --type=merge

6.2 Pod 卡 Terminating

1
2
3
4


kubectl patch pod nacos-0 -n safety-xjprod \
 --type=merge -p '{"metadata":{"finalizers":null}}'

kubectl delete po <pod> -n <ns> --grace-period=0 --force

6.3 后台强删 namespace

1
2


nohup kubectl delete ns cattle-impersonation-system --grace-period=0 --force > /dev/null 2>&1 &
nohup kubectl delete ns cattle-system --grace-period=0 --force > /dev/null 2>&1 &

6.4 KubeSphere 等第三方组件

1
2
3
4
5
6
7
8
9


# 看 CRD
kubectl get crd,clusterrole,clusterrolebinding | grep kubesphere

# 删 cluster CRD
kubectl edit crd clusters.cluster.kubesphere.io
# 删 metadata.finalizers 保存

# 后台删 namespace
nohup kubectl delete namespace kubesphere-system --force --grace-period=0 > /dev/null 2>&1 &

7. 排错速查

现象	原因	解决
升级后 `kubectl get node` 仍是旧版本	部分 master 没升完	检查每台 master 的 `kubelet --version`
`etcdctl: this member is not a leader`	etcd 升级没让 leader 先升	先升 leader，重启会自动选主
`kubectl drain` 报 “cannot delete Pods using local storage”	用了 local PV	加 `--delete-emptydir-data --force`
卸载后 `kubelet` 还在	daemon-reload 没跑	跑 `systemctl daemon-reload` 再 `systemctl reset-failed`
节点还在 cluster 里	`kubectl delete node` 没跑	`kubectl delete node <name>`

8. 小结

升级 / 卸载是 K8s 集群"压箱底"操作：

小版本升级 只换二进制，2 分钟一台
卸载顺序：停服务 → 删单元 → 删目录 → 清环境变量
finalizers 是 namespace 卡死的最常见原因，patch 一下就解决
保留备份（etcd snapshot、admin.kubeconfig）永远不亏

下一步：K8s 节点扩容：worker 加入集群一键脚本。

升级 on Liangweidong's blog