Kube-Apiserver on Liangweidong's blog

K8s 1.28 核心组件二进制部署：apiserver/scheduler/controller-manager

Mon, 15 Apr 2024 00:00:00 +0800

K8s 控制平面三大组件

K8s 控制平面（Control Plane）由 3 个核心组件组成：

kube-apiserver：所有通信的入口，REST API server
kube-scheduler：决定 Pod 调度到哪个 Node
kube-controller-manager：运行所有控制器（Node、Replication、Endpoint、ServiceAccount 等）

本文只讲这 3 个组件的部署与配置，节点组件（kubelet、kube-proxy）单独成篇。

适用版本：Kubernetes 1.28.5 / Ubuntu 22.04 部署位置：master1、master2、master3 各一份

1. 准备目录

1
2
3
4
5


# 所有 master 节点执行
mkdir -p /etc/kubernetes/manifests/ \
 /etc/systemd/system/kubelet.service.d \
 /var/lib/kubelet \
 /var/log/kubernetes

2. K8s 1.28.5 二进制分发

2.1 master1 解压

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


cd /data/softs
tar -xf kubernetes-server-linux-amd64.tar.gz \
 --strip-components=3 \
 -C /usr/local/bin \
 kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}

ls /usr/local/bin
# etcd etcdctl etcdutl kube-apiserver kube-controller-manager kubectl kubelet kube-proxy kube-scheduler

kubelet --version
# Kubernetes v1.28.5

2.2 分发到其他 master

1
2
3
4
5


for NODE in master2 master3; do
 scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} \
 $NODE:/usr/local/bin/
 scp /usr/local/bin/etcd* $NODE:/usr/local/bin/
done

2.3 分发到 worker（只发 kubelet 和 kube-proxy）

1
2
3
4


for NODE in worker1 worker2; do
 echo $NODE
 scp /usr/local/bin/kube{let,-proxy} $NODE:/usr/local/bin/
done

3. K8s 1.28 证书签发

3.1 CA 根证书

1
2
3
4
5
6


mkdir -p /data/softs/pki && cd /data/softs/pki

# CA 根
cfssl gencert -initca ca-csr.json | cfssljson -bare /etc/kubernetes/pki/ca
ls /etc/kubernetes/pki
# ca.csr ca-key.pem ca.pem

3.2 apiserver 证书（关键）

apiserver-csr.json 里的 hosts 字段必须预留未来所有可能加入的 Node IP：

1
2
3
4
5
6
7


cfssl gencert \
 -ca=/etc/kubernetes/pki/ca.pem \
 -ca-key=/etc/kubernetes/pki/ca-key.pem \
 -config=ca-config.json \
 -hostname=10.96.0.1,<vip-internal>,127.0.0.1,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,<master1-ip>,<master2-ip>,<master3-ip> \
 -profile=kubernetes \
 apiserver-csr.json | cfssljson -bare /etc/kubernetes/pki/apiserver

后续扩容加 IP 时，重新生成：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


# 备份旧证书
mkdir -p ~/apiserverbak
cd /etc/kubernetes/pki
mv apiserver.csr apiserver.pem apiserver-key.pem ~/apiserverbak

# 重新签发（加新 IP）
cfssl gencert \
 -ca=/etc/kubernetes/pki/ca.pem \
 -ca-key=/etc/kubernetes/pki/ca-key.pem \
 -config=ca-config.json \
 -hostname=10.96.0.1,<vip-internal>,127.0.0.1,kubernetes,kubernetes.default,...,<新IP1>,<新IP2> \
 -profile=kubernetes \
 apiserver-csr.json | cfssljson -bare /etc/kubernetes/pki/apiserver

# 推送到其他 master
for NODE in master2 master3; do
 for FILE in apiserver.csr apiserver.pem apiserver-key.pem; do
 scp /etc/kubernetes/pki/${FILE} $NODE:/etc/kubernetes/pki/${FILE}
 done
done

# 重启
systemctl restart kube-apiserver.service

3.3 front-proxy 聚合层证书

1
2
3
4
5
6
7


cfssl gencert -initca front-proxy-ca-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-ca
cfssl gencert \
 -ca=/etc/kubernetes/pki/front-proxy-ca.pem \
 -ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem \
 -config=ca-config.json \
 -profile=kubernetes \
 front-proxy-client-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-client

3.4 controller-manager 和 scheduler 证书

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


cfssl gencert \
 -ca=/etc/kubernetes/pki/ca.pem \
 -ca-key=/etc/kubernetes/pki/ca-key.pem \
 -config=ca-config.json \
 -profile=kubernetes \
 manager-csr.json | cfssljson -bare /etc/kubernetes/pki/controller-manager

cfssl gencert \
 -ca=/etc/kubernetes/pki/ca.pem \
 -ca-key=/etc/kubernetes/pki/ca-key.pem \
 -config=ca-config.json \
 -profile=kubernetes \
 scheduler-csr.json | cfssljson -bare /etc/kubernetes/pki/scheduler

# kube-proxy 证书
cfssl gencert \
 -ca=/etc/kubernetes/pki/ca.pem \
 -ca-key=/etc/kubernetes/pki/ca-key.pem \
 -config=ca-config.json \
 -profile=kubernetes \
 kube-proxy-csr.json | cfssljson -bare /etc/kubernetes/pki/kube-proxy

# admin 证书
cfssl gencert \
 -ca=/etc/kubernetes/pki/ca.pem \
 -ca-key=/etc/kubernetes/pki/ca-key.pem \
 -config=ca-config.json \
 -profile=kubernetes \
 admin-csr.json | cfssljson -bare /etc/kubernetes/pki/admin

3.5 ServiceAccount Key（SA 签名密钥对）

1
2


openssl genrsa -out /etc/kubernetes/pki/sa.key 2048
openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/sa.pub

3.6 推送到其他 master

1
2
3
4
5
6
7
8
9


for NODE in master2 master3; do
 ssh $NODE "mkdir -p /etc/kubernetes/pki"
 for FILE in $(ls /etc/kubernetes/pki | grep -v etcd); do
 scp /etc/kubernetes/pki/${FILE} $NODE:/etc/kubernetes/pki/${FILE}
 done
 for FILE in admin.kubeconfig controller-manager.kubeconfig scheduler.kubeconfig; do
 scp /etc/kubernetes/${FILE} $NODE:/etc/kubernetes/${FILE}
 done
done

4. kubeconfig 文件

K8s 用 kubeconfig 描述"集群 + 凭据 + 上下文"三元组。每个组件的 kubeconfig 用 kubectl config set-* 生成。

4.1 controller-manager.kubeconfig

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


kubectl config set-cluster kubernetes \
 --certificate-authority=/etc/kubernetes/pki/ca.pem \
 --embed-certs=true \
 --server=https://<vip-internal>:10443 \
 --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

kubectl config set-credentials system:kube-controller-manager \
 --client-certificate=/etc/kubernetes/pki/controller-manager.pem \
 --client-key=/etc/kubernetes/pki/controller-manager-key.pem \
 --embed-certs=true \
 --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

kubectl config set-context system:kube-controller-manager@kubernetes \
 --cluster=kubernetes \
 --user=system:kube-controller-manager \
 --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

kubectl config use-context system:kube-controller-manager@kubernetes \
 --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

4.2 scheduler.kubeconfig

同上，user 改成 system:kube-scheduler，cert 改成 scheduler.pem。

4.3 admin.kubeconfig

user 改成 kubernetes-admin，cert 改成 admin.pem：

1
2
3
4
5
6
7


kubectl config set-context kubernetes-admin@kubernetes \
 --cluster=kubernetes \
 --user=kubernetes-admin \
 --kubeconfig=/etc/kubernetes/admin.kubeconfig

kubectl config use-context kubernetes-admin@kubernetes \
 --kubeconfig=/etc/kubernetes/admin.kubeconfig

4.4 kube-proxy.kubeconfig

user kube-proxy，cert kube-proxy.pem。

5. kube-apiserver 部署

5.1 kube-apiserver.service（master1）

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44


[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-apiserver \
 --v=2 \
 --allow-privileged=true \
 --bind-address=0.0.0.0 \
 --secure-port=6443 \
 --advertise-address=<master1-ip> \
 --service-cluster-ip-range=10.96.0.0/12,fd00:1111::/112 \
 --service-node-port-range=30000-32767 \
 --etcd-servers=https://<master1-ip>:2379,https://<master2-ip>:2379,https://<master3-ip>:2379 \
 --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \
 --etcd-certfile=/etc/etcd/ssl/etcd.pem \
 --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
 --client-ca-file=/etc/kubernetes/pki/ca.pem \
 --tls-cert-file=/etc/kubernetes/pki/apiserver.pem \
 --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \
 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \
 --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \
 --service-account-key-file=/etc/kubernetes/pki/sa.pub \
 --service-account-signing-key-file=/etc/kubernetes/pki/sa.key \
 --service-account-issuer=https://kubernetes.default.svc.cluster.local \
 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \
 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \
 --authorization-mode=Node,RBAC \
 --enable-bootstrap-token-auth=true \
 --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
 --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \
 --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem \
 --requestheader-allowed-names=aggregator \
 --requestheader-group-headers=X-Remote-Group \
 --requestheader-extra-headers-prefix=X-Remote-Extra- \
 --requestheader-username-headers=X-Remote-User \
 --enable-aggregator-routing=true
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

master2 / master3 的 unit 只需要改 --advertise-address 为各自 IP。

5.2 启动

1
2
3
4
5


systemctl daemon-reload
systemctl enable --now kube-apiserver.service
systemctl restart kube-apiserver.service
systemctl status kube-apiserver.service
journalctl -f -u kube-apiserver

6. kube-controller-manager 部署

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
 --v=2 \
 --bind-address=0.0.0.0 \
 --root-ca-file=/etc/kubernetes/pki/ca.pem \
 --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \
 --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \
 --service-account-private-key-file=/etc/kubernetes/pki/sa.key \
 --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \
 --leader-elect=true \
 --use-service-account-credentials=true \
 --node-monitor-grace-period=40s \
 --node-monitor-period=5s \
 --controllers=*,bootstrapsigner,tokencleaner \
 --allocate-node-cidrs=true \
 --service-cluster-ip-range=10.96.0.0/12,fd00:1111::/112 \
 --cluster-cidr=172.218.0.0/12,fc00:2222::/112 \
 --node-cidr-mask-size-ipv4=24 \
 --node-cidr-mask-size-ipv6=120 \
 --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem
Restart=always
RestartSec=10s

启动：

1
2
3


systemctl daemon-reload
systemctl enable --now kube-controller-manager.service
systemctl restart kube-controller-manager.service

7. kube-scheduler 部署

1
2
3
4
5
6
7
8


[Service]
ExecStart=/usr/local/bin/kube-scheduler \
 --v=2 \
 --bind-address=0.0.0.0 \
 --leader-elect=true \
 --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
Restart=always
RestartSec=10s

8. TLS Bootstrapping（自动签证书给 kubelet）

传统 K8s 部署需要在每个 worker 上手动签 kubelet 证书，1.4 引入的 TLS Bootstrapping 让 worker 用一个临时 token 自动签证书。

8.1 bootstrap-kubelet.kubeconfig

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


kubectl config set-cluster kubernetes \
 --certificate-authority=/etc/kubernetes/pki/ca.pem \
 --embed-certs=true \
 --server=https://<vip-internal>:10443 \
 --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

kubectl config set-credentials tls-bootstrap-token-user \
 --token=<BOOTSTRAP_TOKEN_ID>.<BOOTSTRAP_TOKEN_SECRET> \
 --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

kubectl config set-context tls-bootstrap-token-user@kubernetes \
 --cluster=kubernetes \
 --user=tls-bootstrap-token-user \
 --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

kubectl config use-context tls-bootstrap-token-user@kubernetes \
 --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

Token 形如 c8ad9c.2e4d610cf3e7426e（前半段是 ID，后半段是 Secret），可自定义但要保证长度够。

8.2 bootstrap.secret.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52


apiVersion: v1
kind: Secret
metadata:
 name: bootstrap-token-<BOOTSTRAP_TOKEN_ID>
 namespace: kube-system
type: bootstrap.kubernetes.io/token
stringData:
 description: "The default bootstrap token generated by 'kubelet'."
 token-id: <BOOTSTRAP_TOKEN_ID>
 token-secret: <BOOTSTRAP_TOKEN_SECRET>
 usage-bootstrap-authentication: "true"
 usage-bootstrap-signing: "true"
 auth-extra-groups: system:bootstrappers:default-node-token,system:bootstrappers:worker,system:bootstrappers:ingress
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: kubelet-bootstrap
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: system:node-bootstrapper
subjects:
- apiGroup: rbac.authorization.k8s.io
 kind: Group
 name: system:bootstrappers:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: node-autoapprove-bootstrap
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
 kind: Group
 name: system:bootstrappers:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: node-autoapprove-certificate-rotation
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
 kind: Group
 name: system:nodes

执行：

1
2
3
4
5
6
7
8
9


kubectl create -f bootstrap.secret.yaml

# 验证
kubectl get cs
# Warning: v1 ComponentStatus is deprecated in v1.19+
# NAME STATUS MESSAGE ERROR
# scheduler Healthy ok
# controller-manager Healthy ok
# etcd-0 Healthy ok

8.3 admin 客户端

1
2
3
4


mkdir -p /root/.kube
cp /etc/kubernetes/admin.kubeconfig /root/.kube/config
echo "export KUBECONFIG=/etc/kubernetes/admin.kubeconfig" >> /etc/profile
source /etc/profile

9. 验证

1
2
3


# 所有 master 上
systemctl status kube-apiserver kube-controller-manager kube-scheduler
journalctl -u kube-apiserver --since "5 minutes ago"

3 个 master 都需要启动正常。kubectl get cs 三个组件都应显示 Healthy。

10. 排错

现象	原因	解决
`apiserver: x509: certificate signed by unknown authority`	CA 不一致	同步 `/etc/kubernetes/pki/ca.pem` 到所有 master
`etcd cluster is unavailable`	apiserver 找不到 etcd	检查 `--etcd-servers` + etcd 证书路径
`the server has invalid kubeconfig`	kubeconfig 写错	重新生成
`kubelet TLS bootstrap failed`	token 错或 RBAC 没建	检查 token + `kubectl get clusterrolebinding \| grep bootstrap`
`failed to create listener: address already in use`	6443 端口占用	`ss -tlnp \| grep 6443`

11. 小结

控制平面三大组件是 K8s 的"大脑"。几个关键点：

apiserver 证书必须预留未来 Node IP（避免后续扩容重签）
TLS Bootstrapping 用临时 token省去每个 worker 签证书
3 master 配置必须一致（除了 advertise-address）
kubeconfig 一定要 embed-certs=true避免每次 kubectl 都要指定 CA

下一步：K8s 节点组件部署：kubelet + kube-proxy + IPVS。

K8s API Server 高可用：Nginx 四层 + Keepalived VIP 漂移

Mon, 15 Mar 2021 00:00:00 +0800

写于 2021-03，背景：K8s 1.21 GA 阶段，二进制高可用部署仍然是国内大厂主流。本文用编译版 Nginx 1.24 + Keepalived 2.2 实现 API Server 的统一入口 VIP 漂移。

一、为什么需要 VIP

3 master 集群时，apiserver 监听在 3 个不同 IP:6443：

1
2
3


master1 → 192.168.139.133:6443
master2 → 192.168.139.134:6443
master3 → 192.168.139.135:6443

但 kubectl 只能配 1 个 --server：

1
2


kubectl --server=https://192.168.139.133:6443 get nodes
# 如果 master1 挂了？

解决思路：在 master 节点前再加一层"虚拟 IP"，3 个 master 通过 VRRP 协议竞选 VIP，VIP 永远在 1 个能用的 master 上。

1
2
3
4
5


kubectl → VIP:6443 (192.168.139.150)
 ↓ VRRP
 ┌────┴────┬────────┐
 master1 master2 master3
 :6443 :6443 :6443

二、方案选型

方案	优点	缺点
Nginx + Keepalived	成熟稳定，二层/四层都能做	需要编译
HAProxy + Keepalived	性能比 Nginx 高	配置稍复杂
Envoy + Keepalived	支持高级路由	太重
公有云 SLB	零运维	绑定云厂商

本文选 Nginx stream + Keepalived——最常见、配置清晰。

三、集群规划

主机名	IP	角色
master1	192.168.139.133	nginx + keepalived MASTER
master2	192.168.139.134	nginx + keepalived BACKUP
master3	192.168.139.135	nginx + keepalived BACKUP
VIP	192.168.139.150	漂移地址

端口约定：

apiserver：6443
nginx stream 监听：8443（外部访问 VIP:8443 → 内部 apiserver 6443）
keepalived VRRP：协议号 112

四、编译 Nginx（master1 + master2）

只在两台装 nginx + keepalived 即可（master3 不需要，节省资源）。如果所有 3 台都装，keepalived 仍能正常选举。

4.1 装编译依赖

1

apt install -y make g++ openssl libssl-dev libpcre3 libpcre3-dev zlib1g-dev libgd-dev

4.2 编译 Nginx

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


cd /k8s/softs
tar xf nginx-1.24.0.tar.gz && cd nginx-1.24.0

./configure \
 --prefix=/usr/local/nginx \
 --with-http_ssl_module \
 --with-http_stub_status_module \
 --with-pcre \
 --with-http_gzip_static_module \
 --with-stream \
 --with-stream_ssl_module

make -j$(nproc) && make install

关键参数：**--with-stream**（TCP 四层代理），**--with-stream_ssl_module**（与 apiserver HTTPS 通信）

4.3 Nginx 主配置

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49


cat > /usr/local/nginx/conf/nginx.conf << 'EOF'
worker_processes 1;
events {
 worker_connections 1024;
}
http {
 include mime.types;
 default_type application/octet-stream;
 log_format main '$remote_addr - $remote_user [$time_local] "$request" '
 '$status $body_bytes_sent "$http_referer" '
 '"$http_user_agent" "$http_x_forwarded_for"';

 sendfile on;
 keepalive_timeout 65;
 autoindex on;
 gzip on;
 gzip_buffers 32 4K;
 gzip_comp_level 6;
 gzip_min_length 1k;
 gzip_types application/javascript text/css text/xml;
 gzip_disable "MSIE [1-6]\.";
 gzip_http_version 1.1;
 gzip_vary on;

 client_body_buffer_size 1024k;
 client_max_body_size 2048m;

 client_header_buffer_size 128k;
 large_client_header_buffers 4 128k;

 proxy_buffer_size 256k;
 proxy_buffering on;
 proxy_buffers 64 128k;
 proxy_busy_buffers_size 512k;

 include conf.d/*.conf;
}

stream {
 log_format proxy '[$time_local] $protocol status:$status, '
 'client: $remote_addr, upstream:"$upstream_addr", '
 'bytes: $bytes_sent/$bytes_received '
 'time: $session_time/$upstream_connect_time';

 include stream/*.conf;
}
EOF

mkdir -p /usr/local/nginx/conf/{conf.d,stream}

4.4 stream 配置

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


cat > /usr/local/nginx/conf/stream/stream.conf << 'EOF'
upstream k8sapiserver {
 hash $remote_addr consistent;
 server 192.168.139.133:6443 max_fails=3 fail_timeout=30s;
 server 192.168.139.134:6443 max_fails=3 fail_timeout=30s;
 server 192.168.139.135:6443 max_fails=3 fail_timeout=30s;
}

server {
 error_log logs/8443_error.log;
 access_log logs/8443_access.log proxy;
 listen 8443;
 proxy_connect_timeout 1s;
 proxy_pass k8sapiserver;
}
EOF

关键参数：

hash $remote_addr consistent：同一客户端始终连同一后端（会话保持）
max_fails=3 fail_timeout=30s：3 次失败标记 down，30 秒后重试
proxy_connect_timeout 1s：连不上后端立即失败，让 keepalived 切主

五、systemd 单元

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


[Unit]
Description=kube-apiserver nginx proxy
After=network.target network-online.target
Wants=network-online.target

[Service]
Type=forking
ExecStartPre=/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf -p /usr/local/nginx -t
ExecStart=/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf -p /usr/local/nginx
ExecReload=/bin/kill -s HUP $MAINPID
PrivateTmp=true
Restart=always
RestartSec=5
StartLimitInterval=0
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

启动：

1
2
3
4


systemctl daemon-reload
systemctl enable --now kube-nginx
systemctl status kube-nginx
curl 192.168.139.133:8443 # 不报错即可

六、Keepalived 配置

6.1 编译 Keepalived（master1 + master2）

1
2
3
4
5


cd /k8s/softs
tar xf keepalived-2.2.8.tar.gz && cd keepalived-2.2.8
./configure --prefix=/usr/local/keepalived
make -j$(nproc) && make install
mkdir -p /usr/local/keepalived/conf

6.2 健康检查脚本

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


cat > /usr/local/keepalived/conf/check_nginx.sh << 'EOF'
#!/bin/bash
A=$(ps -C nginx --no-header | wc -l)
if [ $A -eq 0 ]; then
 /usr/bin/systemctl start kube-nginx
 sleep 2
 if [ $(ps -C nginx --no-header | wc -l) -eq 0 ]; then
 /usr/bin/systemctl stop kube-keepalived
 fi
fi
EOF
chmod +x /usr/local/keepalived/conf/check_nginx.sh

逻辑：nginx 挂了 → 尝试重启 → 启不来就停 keepalived，让 VIP 漂到其他节点。

6.3 master1 配置（priority 100 = MASTER）

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


cat > /usr/local/keepalived/conf/kube-keepalived.conf << 'EOF'
global_defs {
 router_id LVS_DEVEL
}
vrrp_script check_nginx {
 script "/usr/local/keepalived/conf/check_nginx.sh"
 interval 5
 weight -5
 fall 2
 rise 1
}
vrrp_instance VI_1 {
 state MASTER
 interface ens33
 mcast_src_ip 192.168.139.133
 virtual_router_id 51
 priority 100
 nopreempt
 advert_int 2
 authentication {
 auth_type PASS
 auth_pass K8SHA_KA_AUTH
 }
 virtual_ipaddress {
 192.168.139.150
 }
 track_script {
 check_nginx
 }
}
EOF

6.4 master2 配置（priority 90 = BACKUP）

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


cat > /usr/local/keepalived/conf/kube-keepalived.conf << 'EOF'
global_defs {
 router_id LVS_DEVEL
}
vrrp_script check_nginx {
 script "/usr/local/keepalived/conf/check_nginx.sh"
 interval 5
 weight -5
 fall 2
 rise 1
}
vrrp_instance VI_1 {
 state BACKUP
 interface ens33
 mcast_src_ip 192.168.139.134
 virtual_router_id 51
 priority 90
 nopreempt
 advert_int 2
 authentication {
 auth_type PASS
 auth_pass K8SHA_KA_AUTH
 }
 virtual_ipaddress {
 192.168.139.150
 }
 track_script {
 check_nginx
 }
}
EOF

关键参数：

virtual_router_id 51：同一集群必须一致（不同集群必须不同！）
priority 100 > 90：数字大 = 高优先级
nopreempt：原 master 恢复后不抢回 VIP（避免脑裂）
track_script：nginx 挂了直接降 priority 5，触发 VIP 漂移
auth_pass K8SHA_KA_AUTH：明文 VRRP 认证，生产请用 16 位随机字符串

6.5 systemd 单元

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


[Unit]
Description=VRRP High Availability Monitor
After=network-online.target syslog.target
Wants=network-online.target
Documentation=https://keepalived.org/manpage.html

[Service]
Type=forking
KillMode=process
ExecStart=/usr/local/keepalived/sbin/keepalived -D -f /usr/local/keepalived/conf/kube-keepalived.conf
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

启动：

1
2
3
4


systemctl daemon-reload
systemctl enable --now kube-keepalived
systemctl status kube-keepalived
journalctl -f -u kube-keepalived

七、验证

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# master1 上看 VIP 是否绑定
ip addr show ens33
# inet 192.168.139.133/24 ...
# inet 192.168.139.150/32 ... ← 看到这行说明 VIP 成功

# 验证 VIP 可访问 apiserver
curl -k https://192.168.139.150:8443/version
# {
# "major": "1",
# "gitVersion": "v1.28.5",
# ...
# }

配置 kubectl 使用 VIP：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


mkdir -p ~/.kube
cat > ~/.kube/config << EOF
apiVersion: v1
kind: Config
clusters:
- cluster:
 server: https://192.168.139.150:8443
 certificate-authority-data: <base64 编码的 ca.pem>
 name: kubernetes
contexts:
- context:
 cluster: kubernetes
 user: kubernetes-admin
 name: default
current-context: default
users:
- name: kubernetes-admin
 user:
 client-certificate-data: <base64 编码的 admin.pem>
 client-key-data: <base64 编码的 admin-key.pem>
EOF

kubectl get nodes

八、故障演练

8.1 模拟 master1 整机宕机

1
2


# 在 master1 上
shutdown -h now

观察 master2：

1
2
3
4
5
6
7


# master2 上
ip addr show ens33
# inet 192.168.139.134/24 ...
# inet 192.168.139.150/32 ... ← VIP 漂到 master2

# 验证 VIP 仍可访问
curl -k https://192.168.139.150:8443/version

预期 5~10 秒后 VIP 漂到 master2，kubectl 无感。

8.2 模拟 master1 nginx 进程挂

1
2


# 在 master1 上
pkill nginx

观察：

check_nginx.sh 检测到 nginx 死 → 尝试 systemctl start kube-nginx
如果能起来：keepalived 继续，VIP 不漂
如果起不来（概率低）：脚本 systemctl stop kube-keepalived，VIP 强制漂走

8.3 恢复 master1

1
2


# master1
systemctl start kube-apiserver kube-nginx kube-keepalived

由于 nopreempt，VIP 不会抢回（继续在 master2）。这是有意设计——避免脑裂。

九、生产环境加固

关闭 mcast_src_ip 改 unicast（云环境 VRRP multicast 被禁用）：

1
2
3
4
5
6
7


vrrp_instance VI_1 {
 ...
 unicast_src_ip 192.168.139.133
 unicast_peer {
 192.168.139.134
 }
}

认证密码用随机 16 位：
1

openssl rand -base64 12
keepalived 配 3 台而不是 2 台：3 节点可以容忍 1 节点挂，2 节点只容忍 0 节点挂（脑裂风险大）。
监控 VIP：把 192.168.139.150 加入 Prometheus blackbox_exporter 监控。
DNS 解析：把 kubernetes-api.example.com 解析到 VIP，kubectl 用域名而不是 IP。

十、常见坑

virtual_router_id 不一致：3 节点 VRRP 选举失败
多播被防火墙挡：VRRP 协议号 112，iptables 默认会拦
nopreempt 没配：master1 恢复时把 VIP 抢回，可能导致短暂脑裂
check_nginx.sh 权限不足：keepalived 跑不了脚本，nginx 挂 VIP 不漂
VIP 绑定到错的 interface：interface ens33 配错网卡，VIP 上不去

十一、前置知识 / 下一步

前置：

nginx 编译与配置
keepalived VRRP 原理
kube-apiserver systemd 单元（参考 2020-12-15《K8s Master 三大组件》）

下一步：

K8s 集群插件（2021-09-15）—— CNI / CoreDNS / Metrics / Dashboard
K8s 集群管理（2021-12-15）—— 升级、节点隔离、排错

参考资料

K8s Master 三大组件安装：apiserver / scheduler / controller-manager

Tue, 15 Dec 2020 00:00:00 +0800

写于 2020-12，背景：K8s 1.20 弃用 dockershim 进入倒计时。本文聚焦 Master 三大组件的参数调优、systemd 守护、与 etcd / kubelet 的协作。

一、Master 三大组件的角色

K8s 控制平面（control plane）由三大组件组成，跑在 master 节点上：

组件	角色	端口
kube-apiserver	集群网关，所有组件都跟它通信	6443 (HTTPS)
kube-scheduler	调度器，决定 Pod 跑哪个节点	无监听端口（HTTP 10251）
kube-controller-manager	控制器集合（30+ 控制器，Deployment/Node/Endpoint…）	无监听端口（HTTP 10252）

协作流程：kubectl → apiserver → scheduler 看到 Pod 没绑定节点 → 选节点写回 apiserver → controller-manager 看到 Pod 被调度 → 创建容器（通过 kubelet）

二、前置准备

所有 master 节点创建目录：

1

mkdir -p /etc/kubernetes/manifests/ /etc/systemd/system/kubelet.service.d /var/lib/kubelet /var/log/kubernetes

前置依赖：

etcd 集群已部署（参考 2020-09-15《etcd 集群部署》）
K8s 证书已签发（参考 2020-03-15《K8s 二进制部署》）

三、kube-apiserver.service

3.1 systemd 单元文件

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44


[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-apiserver \
 --v=2 \
 --allow-privileged=true \
 --bind-address=0.0.0.0 \
 --secure-port=6443 \
 --advertise-address=192.168.139.133 \
 --service-cluster-ip-range=10.96.0.0/12,fd00:1111::/112 \
 --service-node-port-range=30000-32767 \
 --etcd-servers=https://192.168.139.133:2379,https://192.168.139.134:2379,https://192.168.139.135:2379 \
 --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \
 --etcd-certfile=/etc/etcd/ssl/etcd.pem \
 --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
 --client-ca-file=/etc/kubernetes/pki/ca.pem \
 --tls-cert-file=/etc/kubernetes/pki/apiserver.pem \
 --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \
 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \
 --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \
 --service-account-key-file=/etc/kubernetes/pki/sa.pub \
 --service-account-signing-key-file=/etc/kubernetes/pki/sa.key \
 --service-account-issuer=https://kubernetes.default.svc.cluster.local \
 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \
 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \
 --authorization-mode=Node,RBAC \
 --enable-bootstrap-token-auth=true \
 --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
 --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \
 --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem \
 --requestheader-allowed-names=aggregator \
 --requestheader-group-headers=X-Remote-Group \
 --requestheader-extra-headers-prefix=X-Remote-Extra- \
 --requestheader-username-headers=X-Remote-User \
 --enable-aggregator-routing=true
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

3.2 关键参数详解

参数	含义	调优建议
`--advertise-address`	apiserver 对外宣告的 IP	每个 master 不同
`--service-cluster-ip-range`	Service IP 段	/12 给 ~1M Service 够用
`--etcd-servers`	etcd 集群地址	多个用逗号分隔
`--enable-admission-plugins`	准入控制器	至少包含 `NamespaceLifecycle,LimitRanger,ServiceAccount`
`--authorization-mode`	鉴权模式	`Node,RBAC` 是标配
`--requestheader-allowed-names`	聚合层（metrics-server 等）	必须是 `aggregator`
`--enable-bootstrap-token-auth`	kubelet TLS bootstrap	kubeadm 部署必开
`--service-account-issuer`	SA token 签发者	1.20+ 必须配，否则 controller 报错

3.3 启动

1
2
3
4
5


# 所有 master 节点
systemctl daemon-reload
systemctl enable --now kube-apiserver
systemctl status kube-apiserver
journalctl -f -u kube-apiserver

验证（master1）：

1
2
3
4
5
6
7


curl -k https://192.168.139.133:6443/version
# {
# "major": "1",
# "minor": "28",
# "gitVersion": "v1.28.5",
# ...
# }

四、kube-controller-manager.service

4.1 systemd 单元文件

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
 --v=2 \
 --bind-address=0.0.0.0 \
 --root-ca-file=/etc/kubernetes/pki/ca.pem \
 --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \
 --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \
 --service-account-private-key-file=/etc/kubernetes/pki/sa.key \
 --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \
 --leader-elect=true \
 --use-service-account-credentials=true \
 --controllers=*,bootstrapsigner,tokencleaner
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

4.2 关键参数

参数	含义
`--leader-elect=true`	启用 leader 选举，多 master 时只有 leader 干活
`--use-service-account-credentials=true`	每个 controller 用独立 SA（1.20+ 必须）
`--controllers=*`	启用所有 controller（默认），可裁剪
`--cluster-signing-*`	给新节点签发证书用的 CA（kubelet 证书签发）

4.3 controller 分类

kube-controller-manager 实际上是 30+ 控制器的合集：

控制器	作用
Deployment controller	维护 Deployment 期望的副本数
ReplicaSet controller	维护 RS 副本
Node controller	监听节点心跳，标记 NotReady
Endpoint controller	维护 Service ↔ Pod 映射
ServiceAccount controller	自动给 namespace 创建 default SA
Token controller	给 SA 签发 token
Job controller	维护一次性任务
CronJob controller	维护定时任务
PV controller	处理 PV/PVC 绑定
Namespace controller	清理已删除 namespace 的资源
ResourceQuota controller	强制 namespace 资源配额
HorizontalPodAutoscaler	HPA 自动扩缩容
等等…

为什么不裁剪？ K8s 默认 --controllers=*，几乎所有 controller 都需要；只有定制化集群才裁剪（比如不要 serviceaccount-controller，因为用外部 SA 系统）。

五、kube-scheduler.service

5.1 systemd 单元文件

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-scheduler \
 --v=2 \
 --bind-address=0.0.0.0 \
 --leader-elect=true \
 --kubeconfig=/etc/kubernetes/scheduler.kubeconfig \
 --config=/etc/kubernetes/scheduler-config.yaml
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

5.2 调度配置

/etc/kubernetes/scheduler-config.yaml：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
leaderElection:
 leaderElect: true
 resourceNamespace: kube-system
 resourceName: kube-scheduler
profiles:
 - schedulerName: default-scheduler
 plugins:
 score:
 enabled:
 - name: NodeResourcesFit
 - name: NodeAffinity
 - name: PodTopologySpread
 - name: InterPodAffinity
 - name: TaintToleration
 - name: ImageLocality

5.3 调度流程（两阶段）

1. Filtering（过滤）：把所有节点过一遍，留下"能跑"的节点

资源够不够（CPU/内存 request）
端口是否被占用
是否匹配 nodeSelector / nodeAffinity
是否容忍 Pod 的 toleration
是否有 PVC 满足条件

2. Scoring（打分）：对剩下的节点打分，选最高分

资源使用率均衡（避免热点节点）
镜像本地优先（pod 内已有镜像的节点加分）
拓扑打散（Pod 反亲和性）
自定义插件（如 binpack 优化利用率）

六、启动顺序

严格按顺序启动：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# 1. etcd
systemctl start etcd
etcdctl endpoint health --cluster # 验证

# 2. kube-apiserver
systemctl start kube-apiserver
curl -k https://localhost:6443/healthz # 验证

# 3. controller-manager 和 scheduler（可并行）
systemctl start kube-controller-manager
systemctl start kube-scheduler

为什么这个顺序？ apiserver 依赖 etcd；controller-manager 和 scheduler 都通过 apiserver 工作。

七、验证

1
2
3
4
5
6
7
8


# 在 master1 验证
kubectl get componentstatuses
# 或
kubectl get cs
# NAME STATUS MESSAGE ERROR
# controller-manager Healthy ok
# scheduler Healthy ok
# etcd-0 Healthy {"health":"true"}

详细验证：

1
2
3
4
5
6
7
8


# 看 apiserver 性能指标（需 prometheus 或 kubectl proxy）
kubectl get --raw=/metrics | grep apiserver_request_total

# 看 scheduler 调度队列
kubectl get --raw=https://localhost:10251/metrics

# 看 controller-manager 状态
kubectl get --raw=https://localhost:10252/metrics

八、HA 场景下的 leader 选举

3 master 高可用时，scheduler 和 controller-manager 都有 leader 选举机制（apiserver 不用选举，所有 apiserver 都对外服务）：

启动时通过 --leader-elect=true 抢占分布式锁
当前 leader 挂了，其他节点 5~15 秒后接管
kube-scheduler-leader 和 kube-controller-manager-leader 是两个 ConfigMap，存当前 leader 名

查看当前 leader：

1
2


kubectl get cm -n kube-system kube-scheduler-leader -o yaml
kubectl get cm -n kube-system kube-controller-manager-leader -o yaml

九、常见坑

apiserver 启动报 “x509: certificate is not valid for 10.96.0.1”：apiserver 证书的 hostname 列表漏配 10.96.0.1（Service cluster IP）或其他 IP，重新签发
apiserver 启动报 “etcd cluster is unavailable”：先 etcdctl endpoint health 验证 etcd，常见是证书路径错（/etc/etcd/ssl/ vs /etc/kubernetes/pki/etcd/）
controller-manager 启动后 Pod 创建不出 Pod：检查 cluster-signing-cert-file 路径；或 RBAC 权限不足（controller-manager 自己的 SA kubeconfig）
scheduler 日志 “leader election lost”：网络抖动，多 master 情况下短暂无影响
修改配置后不生效：必须 systemctl daemon-reload && systemctl restart kube-apiserver，不是 reload

十、前置知识 / 下一步

前置：

etcd 集群部署（参考 2020-09-15《etcd 集群部署》）
证书签发（参考 2020-03-15《K8s 二进制部署》）

下一步：

API Server 高可用（2021-03-15）—— Nginx 四层 + Keepalived VIP
K8s 集群插件（2021-09-15）—— CNI / CoreDNS / Metrics / Dashboard
K8s 集群管理（2021-12-15）—— 升级、节点隔离