K8s 1.28 核心组件二进制部署：apiserver/scheduler/controller-manager

Mon, 15 Apr 2024 00:00:00 +0800

K8s 控制平面三大组件

K8s 控制平面（Control Plane）由 3 个核心组件组成：

kube-apiserver：所有通信的入口，REST API server
kube-scheduler：决定 Pod 调度到哪个 Node
kube-controller-manager：运行所有控制器（Node、Replication、Endpoint、ServiceAccount 等）

本文只讲这 3 个组件的部署与配置，节点组件（kubelet、kube-proxy）单独成篇。

适用版本：Kubernetes 1.28.5 / Ubuntu 22.04 部署位置：master1、master2、master3 各一份

1. 准备目录

1
2
3
4
5


# 所有 master 节点执行
mkdir -p /etc/kubernetes/manifests/ \
 /etc/systemd/system/kubelet.service.d \
 /var/lib/kubelet \
 /var/log/kubernetes

2. K8s 1.28.5 二进制分发

2.1 master1 解压

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


cd /data/softs
tar -xf kubernetes-server-linux-amd64.tar.gz \
 --strip-components=3 \
 -C /usr/local/bin \
 kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}

ls /usr/local/bin
# etcd etcdctl etcdutl kube-apiserver kube-controller-manager kubectl kubelet kube-proxy kube-scheduler

kubelet --version
# Kubernetes v1.28.5

2.2 分发到其他 master

1
2
3
4
5


for NODE in master2 master3; do
 scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} \
 $NODE:/usr/local/bin/
 scp /usr/local/bin/etcd* $NODE:/usr/local/bin/
done

2.3 分发到 worker（只发 kubelet 和 kube-proxy）

1
2
3
4


for NODE in worker1 worker2; do
 echo $NODE
 scp /usr/local/bin/kube{let,-proxy} $NODE:/usr/local/bin/
done

3. K8s 1.28 证书签发

3.1 CA 根证书

1
2
3
4
5
6


mkdir -p /data/softs/pki && cd /data/softs/pki

# CA 根
cfssl gencert -initca ca-csr.json | cfssljson -bare /etc/kubernetes/pki/ca
ls /etc/kubernetes/pki
# ca.csr ca-key.pem ca.pem

3.2 apiserver 证书（关键）

apiserver-csr.json 里的 hosts 字段必须预留未来所有可能加入的 Node IP：

1
2
3
4
5
6
7


cfssl gencert \
 -ca=/etc/kubernetes/pki/ca.pem \
 -ca-key=/etc/kubernetes/pki/ca-key.pem \
 -config=ca-config.json \
 -hostname=10.96.0.1,<vip-internal>,127.0.0.1,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,<master1-ip>,<master2-ip>,<master3-ip> \
 -profile=kubernetes \
 apiserver-csr.json | cfssljson -bare /etc/kubernetes/pki/apiserver

后续扩容加 IP 时，重新生成：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


# 备份旧证书
mkdir -p ~/apiserverbak
cd /etc/kubernetes/pki
mv apiserver.csr apiserver.pem apiserver-key.pem ~/apiserverbak

# 重新签发（加新 IP）
cfssl gencert \
 -ca=/etc/kubernetes/pki/ca.pem \
 -ca-key=/etc/kubernetes/pki/ca-key.pem \
 -config=ca-config.json \
 -hostname=10.96.0.1,<vip-internal>,127.0.0.1,kubernetes,kubernetes.default,...,<新IP1>,<新IP2> \
 -profile=kubernetes \
 apiserver-csr.json | cfssljson -bare /etc/kubernetes/pki/apiserver

# 推送到其他 master
for NODE in master2 master3; do
 for FILE in apiserver.csr apiserver.pem apiserver-key.pem; do
 scp /etc/kubernetes/pki/${FILE} $NODE:/etc/kubernetes/pki/${FILE}
 done
done

# 重启
systemctl restart kube-apiserver.service

3.3 front-proxy 聚合层证书

1
2
3
4
5
6
7


cfssl gencert -initca front-proxy-ca-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-ca
cfssl gencert \
 -ca=/etc/kubernetes/pki/front-proxy-ca.pem \
 -ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem \
 -config=ca-config.json \
 -profile=kubernetes \
 front-proxy-client-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-client

3.4 controller-manager 和 scheduler 证书

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


cfssl gencert \
 -ca=/etc/kubernetes/pki/ca.pem \
 -ca-key=/etc/kubernetes/pki/ca-key.pem \
 -config=ca-config.json \
 -profile=kubernetes \
 manager-csr.json | cfssljson -bare /etc/kubernetes/pki/controller-manager

cfssl gencert \
 -ca=/etc/kubernetes/pki/ca.pem \
 -ca-key=/etc/kubernetes/pki/ca-key.pem \
 -config=ca-config.json \
 -profile=kubernetes \
 scheduler-csr.json | cfssljson -bare /etc/kubernetes/pki/scheduler

# kube-proxy 证书
cfssl gencert \
 -ca=/etc/kubernetes/pki/ca.pem \
 -ca-key=/etc/kubernetes/pki/ca-key.pem \
 -config=ca-config.json \
 -profile=kubernetes \
 kube-proxy-csr.json | cfssljson -bare /etc/kubernetes/pki/kube-proxy

# admin 证书
cfssl gencert \
 -ca=/etc/kubernetes/pki/ca.pem \
 -ca-key=/etc/kubernetes/pki/ca-key.pem \
 -config=ca-config.json \
 -profile=kubernetes \
 admin-csr.json | cfssljson -bare /etc/kubernetes/pki/admin

3.5 ServiceAccount Key（SA 签名密钥对）

1
2


openssl genrsa -out /etc/kubernetes/pki/sa.key 2048
openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/sa.pub

3.6 推送到其他 master

1
2
3
4
5
6
7
8
9


for NODE in master2 master3; do
 ssh $NODE "mkdir -p /etc/kubernetes/pki"
 for FILE in $(ls /etc/kubernetes/pki | grep -v etcd); do
 scp /etc/kubernetes/pki/${FILE} $NODE:/etc/kubernetes/pki/${FILE}
 done
 for FILE in admin.kubeconfig controller-manager.kubeconfig scheduler.kubeconfig; do
 scp /etc/kubernetes/${FILE} $NODE:/etc/kubernetes/${FILE}
 done
done

4. kubeconfig 文件

K8s 用 kubeconfig 描述"集群 + 凭据 + 上下文"三元组。每个组件的 kubeconfig 用 kubectl config set-* 生成。

4.1 controller-manager.kubeconfig

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


kubectl config set-cluster kubernetes \
 --certificate-authority=/etc/kubernetes/pki/ca.pem \
 --embed-certs=true \
 --server=https://<vip-internal>:10443 \
 --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

kubectl config set-credentials system:kube-controller-manager \
 --client-certificate=/etc/kubernetes/pki/controller-manager.pem \
 --client-key=/etc/kubernetes/pki/controller-manager-key.pem \
 --embed-certs=true \
 --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

kubectl config set-context system:kube-controller-manager@kubernetes \
 --cluster=kubernetes \
 --user=system:kube-controller-manager \
 --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

kubectl config use-context system:kube-controller-manager@kubernetes \
 --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

4.2 scheduler.kubeconfig

同上，user 改成 system:kube-scheduler，cert 改成 scheduler.pem。

4.3 admin.kubeconfig

user 改成 kubernetes-admin，cert 改成 admin.pem：

1
2
3
4
5
6
7


kubectl config set-context kubernetes-admin@kubernetes \
 --cluster=kubernetes \
 --user=kubernetes-admin \
 --kubeconfig=/etc/kubernetes/admin.kubeconfig

kubectl config use-context kubernetes-admin@kubernetes \
 --kubeconfig=/etc/kubernetes/admin.kubeconfig

4.4 kube-proxy.kubeconfig

user kube-proxy，cert kube-proxy.pem。

5. kube-apiserver 部署

5.1 kube-apiserver.service（master1）

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44


[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-apiserver \
 --v=2 \
 --allow-privileged=true \
 --bind-address=0.0.0.0 \
 --secure-port=6443 \
 --advertise-address=<master1-ip> \
 --service-cluster-ip-range=10.96.0.0/12,fd00:1111::/112 \
 --service-node-port-range=30000-32767 \
 --etcd-servers=https://<master1-ip>:2379,https://<master2-ip>:2379,https://<master3-ip>:2379 \
 --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \
 --etcd-certfile=/etc/etcd/ssl/etcd.pem \
 --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
 --client-ca-file=/etc/kubernetes/pki/ca.pem \
 --tls-cert-file=/etc/kubernetes/pki/apiserver.pem \
 --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \
 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \
 --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \
 --service-account-key-file=/etc/kubernetes/pki/sa.pub \
 --service-account-signing-key-file=/etc/kubernetes/pki/sa.key \
 --service-account-issuer=https://kubernetes.default.svc.cluster.local \
 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \
 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \
 --authorization-mode=Node,RBAC \
 --enable-bootstrap-token-auth=true \
 --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
 --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \
 --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem \
 --requestheader-allowed-names=aggregator \
 --requestheader-group-headers=X-Remote-Group \
 --requestheader-extra-headers-prefix=X-Remote-Extra- \
 --requestheader-username-headers=X-Remote-User \
 --enable-aggregator-routing=true
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

master2 / master3 的 unit 只需要改 --advertise-address 为各自 IP。

5.2 启动

1
2
3
4
5


systemctl daemon-reload
systemctl enable --now kube-apiserver.service
systemctl restart kube-apiserver.service
systemctl status kube-apiserver.service
journalctl -f -u kube-apiserver

6. kube-controller-manager 部署

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
 --v=2 \
 --bind-address=0.0.0.0 \
 --root-ca-file=/etc/kubernetes/pki/ca.pem \
 --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \
 --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \
 --service-account-private-key-file=/etc/kubernetes/pki/sa.key \
 --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \
 --leader-elect=true \
 --use-service-account-credentials=true \
 --node-monitor-grace-period=40s \
 --node-monitor-period=5s \
 --controllers=*,bootstrapsigner,tokencleaner \
 --allocate-node-cidrs=true \
 --service-cluster-ip-range=10.96.0.0/12,fd00:1111::/112 \
 --cluster-cidr=172.218.0.0/12,fc00:2222::/112 \
 --node-cidr-mask-size-ipv4=24 \
 --node-cidr-mask-size-ipv6=120 \
 --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem
Restart=always
RestartSec=10s

启动：

1
2
3


systemctl daemon-reload
systemctl enable --now kube-controller-manager.service
systemctl restart kube-controller-manager.service

7. kube-scheduler 部署

1
2
3
4
5
6
7
8


[Service]
ExecStart=/usr/local/bin/kube-scheduler \
 --v=2 \
 --bind-address=0.0.0.0 \
 --leader-elect=true \
 --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
Restart=always
RestartSec=10s

8. TLS Bootstrapping（自动签证书给 kubelet）

传统 K8s 部署需要在每个 worker 上手动签 kubelet 证书，1.4 引入的 TLS Bootstrapping 让 worker 用一个临时 token 自动签证书。

8.1 bootstrap-kubelet.kubeconfig

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


kubectl config set-cluster kubernetes \
 --certificate-authority=/etc/kubernetes/pki/ca.pem \
 --embed-certs=true \
 --server=https://<vip-internal>:10443 \
 --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

kubectl config set-credentials tls-bootstrap-token-user \
 --token=<BOOTSTRAP_TOKEN_ID>.<BOOTSTRAP_TOKEN_SECRET> \
 --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

kubectl config set-context tls-bootstrap-token-user@kubernetes \
 --cluster=kubernetes \
 --user=tls-bootstrap-token-user \
 --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

kubectl config use-context tls-bootstrap-token-user@kubernetes \
 --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

Token 形如 c8ad9c.2e4d610cf3e7426e（前半段是 ID，后半段是 Secret），可自定义但要保证长度够。

8.2 bootstrap.secret.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52


apiVersion: v1
kind: Secret
metadata:
 name: bootstrap-token-<BOOTSTRAP_TOKEN_ID>
 namespace: kube-system
type: bootstrap.kubernetes.io/token
stringData:
 description: "The default bootstrap token generated by 'kubelet'."
 token-id: <BOOTSTRAP_TOKEN_ID>
 token-secret: <BOOTSTRAP_TOKEN_SECRET>
 usage-bootstrap-authentication: "true"
 usage-bootstrap-signing: "true"
 auth-extra-groups: system:bootstrappers:default-node-token,system:bootstrappers:worker,system:bootstrappers:ingress
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: kubelet-bootstrap
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: system:node-bootstrapper
subjects:
- apiGroup: rbac.authorization.k8s.io
 kind: Group
 name: system:bootstrappers:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: node-autoapprove-bootstrap
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
 kind: Group
 name: system:bootstrappers:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: node-autoapprove-certificate-rotation
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
 kind: Group
 name: system:nodes

执行：

1
2
3
4
5
6
7
8
9


kubectl create -f bootstrap.secret.yaml

# 验证
kubectl get cs
# Warning: v1 ComponentStatus is deprecated in v1.19+
# NAME STATUS MESSAGE ERROR
# scheduler Healthy ok
# controller-manager Healthy ok
# etcd-0 Healthy ok

8.3 admin 客户端

1
2
3
4


mkdir -p /root/.kube
cp /etc/kubernetes/admin.kubeconfig /root/.kube/config
echo "export KUBECONFIG=/etc/kubernetes/admin.kubeconfig" >> /etc/profile
source /etc/profile

9. 验证

1
2
3


# 所有 master 上
systemctl status kube-apiserver kube-controller-manager kube-scheduler
journalctl -u kube-apiserver --since "5 minutes ago"

3 个 master 都需要启动正常。kubectl get cs 三个组件都应显示 Healthy。

10. 排错

现象	原因	解决
`apiserver: x509: certificate signed by unknown authority`	CA 不一致	同步 `/etc/kubernetes/pki/ca.pem` 到所有 master
`etcd cluster is unavailable`	apiserver 找不到 etcd	检查 `--etcd-servers` + etcd 证书路径
`the server has invalid kubeconfig`	kubeconfig 写错	重新生成
`kubelet TLS bootstrap failed`	token 错或 RBAC 没建	检查 token + `kubectl get clusterrolebinding \| grep bootstrap`
`failed to create listener: address already in use`	6443 端口占用	`ss -tlnp \| grep 6443`

11. 小结

控制平面三大组件是 K8s 的"大脑"。几个关键点：

apiserver 证书必须预留未来 Node IP（避免后续扩容重签）
TLS Bootstrapping 用临时 token省去每个 worker 签证书
3 master 配置必须一致（除了 advertise-address）
kubeconfig 一定要 embed-certs=true避免每次 kubectl 都要指定 CA

下一步：K8s 节点组件部署：kubelet + kube-proxy + IPVS。

Kubeconfig on Liangweidong's blog