Cfssl on Liangweidong's blog

K8s 二进制部署：软件准备 / 证书 / 工具 / 组件全流程

Thu, 15 May 2025 00:00:00 +0800

写于 2020-03，背景：K8s 1.18 刚 GA，二进制部署仍是国内大厂主流（比 kubeadm 灵活、可定制）。本文以 3 master 高可用集群 + Ubuntu 22.04 为模板，覆盖从软件准备到所有组件 systemd 启动的完整流程。

一、为什么还要学二进制部署

部署方式	优点	缺点
kubeadm	一行命令搞定，官方推荐	黑盒多，问题排查难
二进制	完全可控，组件协作透明	步骤多，容易踩坑
k8s-operator	适合自建 PaaS	上手成本高

结论：生产用 kubeadm 部署，但学习用二进制才能真正理解 K8s。

二、集群规划

3 master + N worker 高可用集群：

主机名	IP	角色
master1	`192.168.139.133`	master + worker（混合）
master2	`192.168.139.134`	master + worker
master3	`192.168.139.135`	master + worker
VIP	`192.168.139.150`	Keepalived 漂移的虚拟 IP
worker1~N	自定	纯 worker

网络规划：

名称	网段
Node 网络	`192.168.139.0/24`
Service 网络	`10.96.0.0/12`
Pod 网络	`172.218.0.0/16`

三、软件准备

所有节点创建统一目录：

1

mkdir -p /k8s/softs

3.1 下载 K8s Server 包

1
2


# 1.28.5 完整 server 包（含所有 master/worker 组件二进制）
wget https://dl.k8s.io/v1.28.5/kubernetes-server-linux-amd64.tar.gz

3.2 下载 etcd

1

wget https://github.com/etcd-io/etcd/releases/download/v3.5.11/etcd-v3.5.11-linux-amd64.tar.gz

3.3 下载 cfssl（证书签发工具）

1
2
3
4
5
6
7
8


wget https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl_1.6.4_linux_amd64
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssljson_1.6.4_linux_amd64
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl-certinfo_1.6.4_linux_amd64

cp cfssl_1.6.4_linux_amd64 /usr/local/bin/cfssl
cp cfssljson_1.6.4_linux_amd64 /usr/local/bin/cfssljson
cp cfssl-certinfo_1.6.4_linux_amd64 /usr/local/bin/cfssl-certinfo
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo

四、系统配置（所有节点）

4.1 禁用 swap

1
2
3


free -m
swapoff -a
sed -ri 's/.*swap.*/#&/' /etc/fstab

4.2 禁用防火墙

1
2
3
4
5
6


# Ubuntu
sudo systemctl stop ufw
sudo systemctl disable ufw
# CentOS 7
systemctl stop firewalld
systemctl disable firewalld

4.3 关闭 SELinux（CentOS）

1
2


setenforce 0
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config

4.4 时间同步

1
2
3


timedatectl set-timezone Asia/Shanghai
apt install -y ntpdate
echo "0 */1 * * * ntpdate time1.aliyun.com" >> /var/spool/cron/crontabs/root

4.5 加载 IPVS 模块

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


apt install ipvsadm ipset sysstat conntrack -y
cat >> /etc/modules-load.d/ipvs.conf <<EOF
ip_vs
ip_vs_rr
ip_vs_wwr
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
EOF
systemctl restart systemd-modules-load.service
lsmod | grep -e ip_vs -e nf_conntrack

4.6 内核参数

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


cat << EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory = 1
vm.panic_on_oom = 0
fs.inotify.max_user_watches = 89100
fs.file-max = 52706963
fs.nr_open = 52706963
net.netfilter.nf_conntrack_max = 2310720
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl = 15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.core.somaxconn = 16384
EOF
sysctl --system

4.7 ulimit 调优

1
2
3
4
5
6
7
8
9


ulimit -SHn 65535
cat >> /etc/security/limits.conf <<EOF
* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited
EOF

五、证书签发（master1 操作）

K8s 集群里几乎所有通信都走 TLS，证书是入门第一道坎。

5.1 etcd 证书

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


mkdir -p /k8s/softs/pki && cd /k8s/softs/pki

# 写入 ca 配置
cat > ca-config.json << EOF
{
 "signing": {
 "default": { "expiry": "876000h" },
 "profiles": {
 "kubernetes": {
 "usages": ["signing", "key encipherment", "server auth", "client auth"],
 "expiry": "876000h"
 }
 }
 }
}
EOF

# etcd CA
cat > etcd-ca-csr.json << EOF
{
 "CN": "etcd",
 "key": { "algo": "rsa", "size": 2048 },
 "names": [{ "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "etcd", "OU": "Etcd Security" }],
 "ca": { "expiry": "876000h" }
}
EOF

cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare /etc/etcd/ssl/etcd-ca

# etcd server 证书（注意 hostname 和 IP）
cat > etcd-csr.json << EOF
{
 "CN": "etcd",
 "key": { "algo": "rsa", "size": 2048 },
 "names": [{ "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "etcd", "OU": "Etcd Security" }]
}
EOF

cfssl gencert \
 -ca=/etc/etcd/ssl/etcd-ca.pem \
 -ca-key=/etc/etcd/ssl/etcd-ca-key.pem \
 -config=ca-config.json \
 -hostname=127.0.0.1,master1,master2,master3,192.168.139.133,192.168.139.134,192.168.139.135 \
 -profile=kubernetes \
 etcd-csr.json | cfssljson -bare /etc/etcd/ssl/etcd

5.2 K8s CA 与 apiserver 证书

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


# K8s CA（10 年有效）
cat > ca-csr.json << EOF
{
 "CN": "kubernetes",
 "key": { "algo": "rsa", "size": 2048 },
 "names": [{ "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "kubernetes", "OU": "K8s CA" }],
 "ca": { "expiry": "876000h" }
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare /etc/kubernetes/pki/ca

# apiserver 证书（多个 hostname/IP）
cat > apiserver-csr.json << EOF
{
 "CN": "kube-apiserver",
 "key": { "algo": "rsa", "size": 2048 },
 "names": [{ "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "kubernetes", "OU": "K8s" }]
}
EOF

cfssl gencert \
 -ca=/etc/kubernetes/pki/ca.pem \
 -ca-key=/etc/kubernetes/pki/ca-key.pem \
 -config=ca-config.json \
 -hostname=10.96.0.1,192.168.139.133,192.168.139.134,192.168.139.135,192.168.139.150,127.0.0.1,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local \
 -profile=kubernetes \
 apiserver-csr.json | cfssljson -bare /etc/kubernetes/pki/apiserver

5.3 前置代理证书（aggregator）

1
2
3
4
5
6
7
8


# front-proxy CA + client
cfssl gencert -initca front-proxy-ca-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-ca
cfssl gencert \
 -ca=/etc/kubernetes/pki/front-proxy-ca.pem \
 -ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem \
 -config=ca-config.json \
 -profile=kubernetes front-proxy-client-csr.json | \
 cfssljson -bare /etc/kubernetes/pki/front-proxy-client

5.4 controller-manager / scheduler / admin / kube-proxy 证书

按角色签发，证书 CN 决定 K8s RBAC 里的身份（system:kube-controller-manager 等）：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


# controller-manager
cfssl gencert -ca=/etc/kubernetes/pki/ca.pem \
 -ca-key=/etc/kubernetes/pki/ca-key.pem -config=ca-config.json \
 -profile=kubernetes manager-csr.json | \
 cfssljson -bare /etc/kubernetes/pki/controller-manager

# scheduler
cfssl gencert -ca=/etc/kubernetes/pki/ca.pem \
 -ca-key=/etc/kubernetes/pki/ca-key.pem -config=ca-config.json \
 -profile=kubernetes scheduler-csr.json | \
 cfssljson -bare /etc/kubernetes/pki/scheduler

# admin
cfssl gencert -ca=/etc/kubernetes/pki/ca.pem \
 -ca-key=/etc/kubernetes/pki/ca-key.pem -config=ca-config.json \
 -profile=kubernetes admin-csr.json | \
 cfssljson -bare /etc/kubernetes/pki/admin

# kube-proxy
cfssl gencert -ca=/etc/kubernetes/pki/ca.pem \
 -ca-key=/etc/kubernetes/pki/ca-key.pem -config=ca-config.json \
 -profile=kubernetes kube-proxy-csr.json | \
 cfssljson -bare /etc/kubernetes/pki/kube-proxy

5.5 SA（ServiceAccount）密钥对

1
2


openssl genrsa -out /etc/kubernetes/pki/sa.key 2048
openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/sa.pub

5.6 同步证书到其他 master

1
2
3
4
5
6


for NODE in master2 master3; do
 ssh $NODE "mkdir -p /etc/kubernetes/pki /etc/etcd/ssl"
 for DIR in etcd/ssl kubernetes/pki; do
 scp -r /etc/$DIR/* $NODE:/etc/$DIR/
 done
done

六、生成 kubeconfig（每个组件一个身份）

kubeconfig 包含集群地址 + 用户证书 + 上下文：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


KUBECONFIG_FILE=/etc/kubernetes/admin.kubeconfig
kubectl config set-cluster kubernetes \
 --certificate-authority=/etc/kubernetes/pki/ca.pem \
 --embed-certs=true --server=https://192.168.139.150:6443 \
 --kubeconfig=$KUBECONFIG_FILE

kubectl config set-credentials kubernetes-admin \
 --client-certificate=/etc/kubernetes/pki/admin.pem \
 --client-key=/etc/kubernetes/pki/admin-key.pem \
 --embed-certs=true \
 --kubeconfig=$KUBECONFIG_FILE

kubectl config set-context kubernetes-admin@kubernetes \
 --cluster=kubernetes --user=kubernetes-admin \
 --kubeconfig=$KUBECONFIG_FILE

kubectl config use-context kubernetes-admin@kubernetes --kubeconfig=$KUBECONFIG_FILE

controller-manager / scheduler / kube-proxy 各自生成一份 kubeconfig。

七、解压二进制到 /usr/local/bin

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# K8s 组件（master + worker 都有）
cd /k8s/softs
tar -xf kubernetes-server-linux-amd64.tar.gz \
 --strip-components=3 -C /usr/local/bin \
 kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}

# etcd
tar -xf etcd-v3.5.11-linux-amd64.tar.gz
mv etcd-v3.5.11-linux-amd64/etcd /usr/local/bin/
mv etcd-v3.5.11-linux-amd64/etcdctl /usr/local/bin/
mv etcd-v3.5.11-linux-amd64/etcdutl /usr/local/bin/

同步到其他节点：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# master 节点
for NODE in master2 master3; do
 scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} $NODE:/usr/local/bin/
 scp /usr/local/bin/etcd* $NODE:/usr/local/bin/
done

# worker 节点
for NODE in worker1 worker2; do
 scp /usr/local/bin/kube{let,-proxy} $NODE:/usr/local/bin/
done

八、systemd 单元文件

8.1 kube-apiserver.service

所有 master 节点都需要，注意 --advertise-address 每个 master 不同：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43


[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-apiserver \
 --v=2 \
 --allow-privileged=true \
 --bind-address=0.0.0.0 \
 --secure-port=6443 \
 --advertise-address=192.168.139.133 \
 --service-cluster-ip-range=10.96.0.0/12,fd00:1111::/112 \
 --service-node-port-range=30000-32767 \
 --etcd-servers=https://192.168.139.133:2379,https://192.168.139.134:2379,https://192.168.139.135:2379 \
 --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \
 --etcd-certfile=/etc/etcd/ssl/etcd.pem \
 --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
 --client-ca-file=/etc/kubernetes/pki/ca.pem \
 --tls-cert-file=/etc/kubernetes/pki/apiserver.pem \
 --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \
 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \
 --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \
 --service-account-key-file=/etc/kubernetes/pki/sa.pub \
 --service-account-signing-key-file=/etc/kubernetes/pki/sa.key \
 --service-account-issuer=https://kubernetes.default.svc.cluster.local \
 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \
 --authorization-mode=Node,RBAC \
 --enable-bootstrap-token-auth=true \
 --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
 --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \
 --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem \
 --requestheader-allowed-names=aggregator \
 --requestheader-group-headers=X-Remote-Group \
 --requestheader-extra-headers-prefix=X-Remote-Extra- \
 --requestheader-username-headers=X-Remote-User \
 --enable-aggregator-routing=true
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

8.2 kube-controller-manager.service

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
 --v=2 \
 --bind-address=0.0.0.0 \
 --root-ca-file=/etc/kubernetes/pki/ca.pem \
 --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \
 --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \
 --service-account-private-key-file=/etc/kubernetes/pki/sa.key \
 --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \
 --leader-elect=true \
 --use-service-account-credentials=true
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

8.3 kube-scheduler.service

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-scheduler \
 --v=2 \
 --bind-address=0.0.0.0 \
 --leader-elect=true \
 --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

8.4 kubelet.service（所有 master + worker 节点）

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kubelet \
 --v=2 \
 --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig \
 --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
 --config=/etc/kubernetes/kubelet-config.yaml \
 --cert-dir=/etc/kubernetes/pki \
 --hostname-override=<本节点主机名>
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

8.5 kube-proxy.service

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-proxy \
 --v=2 \
 --config=/etc/kubernetes/kube-proxy-config.yaml \
 --hostname-override=<本节点主机名>
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

九、启动顺序与验证

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# 所有 master 节点
systemctl daemon-reload
systemctl enable --now kube-apiserver kube-controller-manager kube-scheduler kubelet kube-proxy
systemctl status kube-apiserver

# 验证（master1）
kubectl get componentstatuses
# NAME STATUS MESSAGE ERROR
# controller-manager Healthy ok
# scheduler Healthy ok
# etcd-0 Healthy {"health":"true"}

十、常见坑

证书 hostname 漏配：apiserver 报 “x509: certificate is not valid for any names”，重新签发并把 VIP、Service IP、k8s 域名都加进去
bootstrap token 过期：默认 24h，过期后 worker 无法 join，kubeadm token create --print-join-command 重新生成（kubeadm 部署）
etcd 启动后 apiserver 连不上：etcd 证书路径不对，配置里写 /etc/etcd/ssl/，但 cert 实际在 /etc/kubernetes/pki/etcd/，加软链
kubelet 起不来：bootstrap-kubelet.kubeconfig 不存在或 token 失效
kube-proxy 模式选错：默认 iptables，高并发场景（>5000 Service）改 IPVS 性能更好

十一、前置知识 / 下一步

前置：

K8s 架构概念（参考 2017-06-15《Kubernetes 入门》）
etcd 基本原理（参考 2020-09-15《etcd 集群部署》）
Linux 网络基础（iptables / IPVS / 网桥）

下一步：

etcd 集群部署（2020-09-15）—— K8s 数据层高可用
API Server 高可用（2021-03-15）—— Nginx 四层 + Keepalived VIP
K8s 集群插件（2021-09-15）—— CNI / CoreDNS / Metrics / Dashboard

参考资料

Kubernetes The Hard Way - Kelsey Hightower
Kubernetes 官方文档 - kubeadm 安装
cfssl 官方文档
Kubernetes 组件参数参考

etcd 3.5 集群二进制部署与 TLS 证书签发

Fri, 15 Mar 2024 00:00:00 +0800

为什么 etcd 必须独立部署

etcd 是 K8s 唯一的状态存储后端，所有 Pod/Service/ConfigMap 状态都写在 etcd 里。生产环境对 etcd 的要求：

3 节点或 5 节点集群（raft 多数派容错）
所有通信必须 TLS（peer + client）
必须与 kube-apiserver 分开部署（不要复用 master 节点或用容器化 etcd）

本文用 cfssl 工具签发全套证书，部署 3 节点 etcd 集群。

适用版本：etcd v3.5.11 / cfssl 1.6.4 / Ubuntu 22.04 部署位置：master1、master2、master3 各一份

1. 安装 cfssl 工具

1
2
3
4
5
6
7
8


cd /data/softs
cp cfssl_1.6.4_linux_amd64 /usr/local/bin/cfssl
cp cfssljson_1.6.4_linux_amd64 /usr/local/bin/cfssljson
cp cfssl-certinfo_1.6.4_linux_amd64 /usr/local/bin/cfssl-certinfo
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo

ls /usr/local/bin
# cfssl cfssl-certinfo cfssljson containerd ...

2. etcd 证书生成

所有证书的生成操作只在 master1 上做，然后 scp 到其他 master。

2.1 创建证书目录

1
2


mkdir -p /etc/etcd/ssl
mkdir -p /data/softs/pki && cd /data/softs/pki

需要的 11 个 JSON 配置文件（生产里通常从运维仓库分发）：

1
2
3
4


admin-csr.json apiserver-csr.json ca-config.json
ca-csr.json etcd-ca-csr.json etcd-csr.json
front-proxy-ca-csr.json front-proxy-client-csr.json
kube-proxy-csr.json manager-csr.json scheduler-csr.json

2.2 生成 etcd CA 与证书

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# 生成 CA 根证书
cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare /etc/etcd/ssl/etcd-ca

# 生成 etcd server 证书（hostname 和 IP 必须包含所有 3 个节点）
cfssl gencert \
 -ca=/etc/etcd/ssl/etcd-ca.pem \
 -ca-key=/etc/etcd/ssl/etcd-ca-key.pem \
 -config=ca-config.json \
 -hostname=127.0.0.1,master1,master2,master3,<master1-ip>,<master2-ip>,<master3-ip> \
 -profile=kubernetes \
 etcd-csr.json | cfssljson -bare /etc/etcd/ssl/etcd

生成结果：

1
2
3


ls /etc/etcd/ssl/
# etcd-ca.csr etcd-ca-key.pem etcd-ca.pem
# etcd.csr etcd-key.pem etcd.pem

2.3 把证书复制到其他 master

1
2
3
4
5
6


for NODE in master2 master3; do
 ssh $NODE "mkdir -p /etc/etcd/ssl"
 for FILE in etcd-ca-key.pem etcd-ca.pem etcd-key.pem etcd.pem; do
 scp /etc/etcd/ssl/${FILE} $NODE:/etc/etcd/ssl/${FILE}
 done
done

内网 IP 用占位符 <master1-ip> 等替代；实际操作换成真实 IP。

3. etcd 二进制安装

1
2
3
4
5
6
7
8
9


cd /data/softs
tar -xf etcd-v3.5.11-linux-amd64.tar.gz
mv etcd-v3.5.11-linux-amd64/etcd /usr/local/bin/
mv etcd-v3.5.11-linux-amd64/etcdctl /usr/local/bin/
mv etcd-v3.5.11-linux-amd64/etcdutl /usr/local/bin/

etcdctl version
# etcdctl version: 3.5.11
# API version: 3.5

推送到 master2 / master3：

1
2
3


for NODE in master2 master3; do
 scp /usr/local/bin/etcd* $NODE:/usr/local/bin/
done

4. etcd 配置文件

4.1 etcd.config.yml（master1 示例）

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


name: 'master1'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://<master1-ip>:2380'
listen-client-urls: 'https://<master1-ip>:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
initial-advertise-peer-urls: 'https://<master1-ip>:2380'
advertise-client-urls: 'https://<master1-ip>:2379'
initial-cluster: 'master1=https://<master1-ip>:2380,master2=https://<master2-ip>:2380,master3=https://<master3-ip>:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
client-transport-security:
 cert-file: '/etc/etcd/ssl/etcd.pem'
 key-file: '/etc/etcd/ssl/etcd-key.pem'
 client-cert-auth: true
 trusted-ca-file: '/etc/etcd/ssl/etcd-ca.pem'
 auto-tls: true
peer-transport-security:
 cert-file: '/etc/etcd/ssl/etcd.pem'
 key-file: '/etc/etcd/ssl/etcd-key.pem'
 peer-client-cert-auth: true
 trusted-ca-file: '/etc/etcd/ssl/etcd-ca.pem'
 auto-tls: true

master2 / master3 的配置只需要改 name 字段、listen-peer-urls 里的 IP。

4.2 etcd.service 单元

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


[Unit]
Description=Etcd Service
Documentation=https://coreos.com/etcd/docs/latest/
After=network.target

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/etcd.config.yml
Restart=on-failure
RestartSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
Alias=etcd3.service

5. 启动与验证

5.1 启动

1
2
3


systemctl daemon-reload
systemctl enable --now etcd.service
systemctl status etcd.service

5.2 验证集群状态

1
2
3
4
5
6
7


export ETCDCTL_API=3

etcdctl --endpoints="<master1-ip>:2379,<master2-ip>:2379,<master3-ip>:2379" \
 --cacert=/etc/etcd/ssl/etcd-ca.pem \
 --cert=/etc/etcd/ssl/etcd.pem \
 --key=/etc/etcd/ssl/etcd-key.pem \
 endpoint status --write-out=table

预期输出：

1
2
3
4
5
6
7


+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| <master1-ip>:2379 | b55511e3290daa2b | 3.5.11 | 20 kB | true | false | 2 | 8 | 8 | |
| <master2-ip>:2379 | 64aba84b878707bb | 3.5.11 | 20 kB | false | false | 2 | 8 | 8 | |
| <master3-ip>:2379 | 3eafefedb1462a99 | 3.5.11 | 20 kB | false | false | 2 | 8 | 8 | |
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

只要 IS LEADER 有一个 true，RAFT TERM 都一致，ERRORS 都为空，集群就是健康的。

5.3 etcd 性能压测

1

etcdctl check perf

5.4 备份与恢复

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# 备份
etcdctl snapshot save /tmp/etcd-snapshot.db
etcdctl snapshot status /tmp/etcd-snapshot.db --write-out=table

# 恢复（停 etcd 后）
etcdctl snapshot restore /tmp/etcd-snapshot.db \
 --name master1 \
 --initial-cluster master1=https://...,master2=...,master3=... \
 --initial-advertise-peer-urls https://<master1-ip>:2380 \
 --data-dir /var/lib/etcd-restore

6. 排错速查

现象	原因	解决
`etcdctl: command not found`	旧版本客户端	`export ETCDCTL_API=3`
`certificate verify failed`	CA 不匹配	检查 `trusted-ca-file`
raft term 不增长	集群未选主	检查 `initial-cluster` 是否 3 节点都在
`db size exceeds quota`	默认 2GB	调大 `quota-backend-bytes`
peer URL 改后无法加入	老的 peer 还在	删 data-dir 重启

7. 小结

etcd 集群是 K8s 状态的"心脏"：

必须 3 节点或 5 节点（raft 多数派）
所有通信必须 TLS（peer + client 都要）
证书 hostname 列表必须包含所有 3 节点
定期备份（etcdctl snapshot save）

下一步：K8s 1.28 核心组件二进制部署：apiserver/scheduler/controller-manager。