Featured image of post K8s 二进制部署:软件准备 / 证书 / 工具 / 组件全流程

K8s 二进制部署:软件准备 / 证书 / 工具 / 组件全流程

二进制部署是理解 K8s 集群内部协作的最佳方式。本文覆盖离线软件包、cfssl 证书签发、kube-apiserver/scheduler/controller-manager/kubelet/kube-proxy 全套 systemd 单元,K8s 1.28 实战。

写于 2020-03,背景:K8s 1.18 刚 GA,二进制部署仍是国内大厂主流(比 kubeadm 灵活、可定制)。本文以 3 master 高可用集群 + Ubuntu 22.04 为模板,覆盖从软件准备到所有组件 systemd 启动的完整流程。

一、为什么还要学二进制部署

部署方式优点缺点
kubeadm一行命令搞定,官方推荐黑盒多,问题排查难
二进制完全可控,组件协作透明步骤多,容易踩坑
k8s-operator适合自建 PaaS上手成本高

结论:生产用 kubeadm 部署,但学习用二进制才能真正理解 K8s。

二、集群规划

3 master + N worker 高可用集群:

主机名IP角色
master1192.168.139.133master + worker(混合)
master2192.168.139.134master + worker
master3192.168.139.135master + worker
VIP192.168.139.150Keepalived 漂移的虚拟 IP
worker1~N自定纯 worker

网络规划

名称网段
Node 网络192.168.139.0/24
Service 网络10.96.0.0/12
Pod 网络172.218.0.0/16

三、软件准备

所有节点创建统一目录:

1
mkdir -p /k8s/softs

3.1 下载 K8s Server 包

1
2
# 1.28.5 完整 server 包(含所有 master/worker 组件二进制)
wget https://dl.k8s.io/v1.28.5/kubernetes-server-linux-amd64.tar.gz

3.2 下载 etcd

1
wget https://github.com/etcd-io/etcd/releases/download/v3.5.11/etcd-v3.5.11-linux-amd64.tar.gz

3.3 下载 cfssl(证书签发工具)

1
2
3
4
5
6
7
8
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl_1.6.4_linux_amd64
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssljson_1.6.4_linux_amd64
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl-certinfo_1.6.4_linux_amd64

cp cfssl_1.6.4_linux_amd64 /usr/local/bin/cfssl
cp cfssljson_1.6.4_linux_amd64 /usr/local/bin/cfssljson
cp cfssl-certinfo_1.6.4_linux_amd64 /usr/local/bin/cfssl-certinfo
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo

四、系统配置(所有节点)

4.1 禁用 swap

1
2
3
free -m
swapoff -a
sed -ri 's/.*swap.*/#&/' /etc/fstab

4.2 禁用防火墙

1
2
3
4
5
6
# Ubuntu
sudo systemctl stop ufw
sudo systemctl disable ufw
# CentOS 7
systemctl stop firewalld
systemctl disable firewalld

4.3 关闭 SELinux(CentOS)

1
2
setenforce 0
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config

4.4 时间同步

1
2
3
timedatectl set-timezone Asia/Shanghai
apt install -y ntpdate
echo "0 */1 * * * ntpdate time1.aliyun.com" >> /var/spool/cron/crontabs/root

4.5 加载 IPVS 模块

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
apt install ipvsadm ipset sysstat conntrack -y
cat >> /etc/modules-load.d/ipvs.conf <<EOF
ip_vs
ip_vs_rr
ip_vs_wwr
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
EOF
systemctl restart systemd-modules-load.service
lsmod | grep -e ip_vs -e nf_conntrack

4.6 内核参数

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
cat << EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory = 1
vm.panic_on_oom = 0
fs.inotify.max_user_watches = 89100
fs.file-max = 52706963
fs.nr_open = 52706963
net.netfilter.nf_conntrack_max = 2310720
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl = 15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.core.somaxconn = 16384
EOF
sysctl --system

4.7 ulimit 调优

1
2
3
4
5
6
7
8
9
ulimit -SHn 65535
cat >> /etc/security/limits.conf <<EOF
* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited
EOF

五、证书签发(master1 操作)

K8s 集群里几乎所有通信都走 TLS,证书是入门第一道坎。

5.1 etcd 证书

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
mkdir -p /k8s/softs/pki && cd /k8s/softs/pki

# 写入 ca 配置
cat > ca-config.json << EOF
{
  "signing": {
    "default": { "expiry": "876000h" },
    "profiles": {
      "kubernetes": {
        "usages": ["signing", "key encipherment", "server auth", "client auth"],
        "expiry": "876000h"
      }
    }
  }
}
EOF

# etcd CA
cat > etcd-ca-csr.json << EOF
{
  "CN": "etcd",
  "key": { "algo": "rsa", "size": 2048 },
  "names": [{ "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "etcd", "OU": "Etcd Security" }],
  "ca": { "expiry": "876000h" }
}
EOF

cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare /etc/etcd/ssl/etcd-ca

# etcd server 证书(注意 hostname 和 IP)
cat > etcd-csr.json << EOF
{
  "CN": "etcd",
  "key": { "algo": "rsa", "size": 2048 },
  "names": [{ "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "etcd", "OU": "Etcd Security" }]
}
EOF

cfssl gencert \
  -ca=/etc/etcd/ssl/etcd-ca.pem \
  -ca-key=/etc/etcd/ssl/etcd-ca-key.pem \
  -config=ca-config.json \
  -hostname=127.0.0.1,master1,master2,master3,192.168.139.133,192.168.139.134,192.168.139.135 \
  -profile=kubernetes \
  etcd-csr.json | cfssljson -bare /etc/etcd/ssl/etcd

5.2 K8s CA 与 apiserver 证书

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
# K8s CA(10 年有效)
cat > ca-csr.json << EOF
{
  "CN": "kubernetes",
  "key": { "algo": "rsa", "size": 2048 },
  "names": [{ "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "kubernetes", "OU": "K8s CA" }],
  "ca": { "expiry": "876000h" }
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare /etc/kubernetes/pki/ca

# apiserver 证书(多个 hostname/IP)
cat > apiserver-csr.json << EOF
{
  "CN": "kube-apiserver",
  "key": { "algo": "rsa", "size": 2048 },
  "names": [{ "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "kubernetes", "OU": "K8s" }]
}
EOF

cfssl gencert \
  -ca=/etc/kubernetes/pki/ca.pem \
  -ca-key=/etc/kubernetes/pki/ca-key.pem \
  -config=ca-config.json \
  -hostname=10.96.0.1,192.168.139.133,192.168.139.134,192.168.139.135,192.168.139.150,127.0.0.1,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local \
  -profile=kubernetes \
  apiserver-csr.json | cfssljson -bare /etc/kubernetes/pki/apiserver

5.3 前置代理证书(aggregator)

1
2
3
4
5
6
7
8
# front-proxy CA + client
cfssl gencert -initca front-proxy-ca-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-ca
cfssl gencert \
  -ca=/etc/kubernetes/pki/front-proxy-ca.pem \
  -ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes front-proxy-client-csr.json | \
  cfssljson -bare /etc/kubernetes/pki/front-proxy-client

5.4 controller-manager / scheduler / admin / kube-proxy 证书

按角色签发,证书 CN 决定 K8s RBAC 里的身份(system:kube-controller-manager 等):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# controller-manager
cfssl gencert -ca=/etc/kubernetes/pki/ca.pem \
  -ca-key=/etc/kubernetes/pki/ca-key.pem -config=ca-config.json \
  -profile=kubernetes manager-csr.json | \
  cfssljson -bare /etc/kubernetes/pki/controller-manager

# scheduler
cfssl gencert -ca=/etc/kubernetes/pki/ca.pem \
  -ca-key=/etc/kubernetes/pki/ca-key.pem -config=ca-config.json \
  -profile=kubernetes scheduler-csr.json | \
  cfssljson -bare /etc/kubernetes/pki/scheduler

# admin
cfssl gencert -ca=/etc/kubernetes/pki/ca.pem \
  -ca-key=/etc/kubernetes/pki/ca-key.pem -config=ca-config.json \
  -profile=kubernetes admin-csr.json | \
  cfssljson -bare /etc/kubernetes/pki/admin

# kube-proxy
cfssl gencert -ca=/etc/kubernetes/pki/ca.pem \
  -ca-key=/etc/kubernetes/pki/ca-key.pem -config=ca-config.json \
  -profile=kubernetes kube-proxy-csr.json | \
  cfssljson -bare /etc/kubernetes/pki/kube-proxy

5.5 SA(ServiceAccount)密钥对

1
2
openssl genrsa -out /etc/kubernetes/pki/sa.key 2048
openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/sa.pub

5.6 同步证书到其他 master

1
2
3
4
5
6
for NODE in master2 master3; do
  ssh $NODE "mkdir -p /etc/kubernetes/pki /etc/etcd/ssl"
  for DIR in etcd/ssl kubernetes/pki; do
    scp -r /etc/$DIR/* $NODE:/etc/$DIR/
  done
done

六、生成 kubeconfig(每个组件一个身份)

kubeconfig 包含集群地址 + 用户证书 + 上下文:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
KUBECONFIG_FILE=/etc/kubernetes/admin.kubeconfig
kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/pki/ca.pem \
  --embed-certs=true --server=https://192.168.139.150:6443 \
  --kubeconfig=$KUBECONFIG_FILE

kubectl config set-credentials kubernetes-admin \
  --client-certificate=/etc/kubernetes/pki/admin.pem \
  --client-key=/etc/kubernetes/pki/admin-key.pem \
  --embed-certs=true \
  --kubeconfig=$KUBECONFIG_FILE

kubectl config set-context kubernetes-admin@kubernetes \
  --cluster=kubernetes --user=kubernetes-admin \
  --kubeconfig=$KUBECONFIG_FILE

kubectl config use-context kubernetes-admin@kubernetes --kubeconfig=$KUBECONFIG_FILE

controller-manager / scheduler / kube-proxy 各自生成一份 kubeconfig。

七、解压二进制到 /usr/local/bin

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# K8s 组件(master + worker 都有)
cd /k8s/softs
tar -xf kubernetes-server-linux-amd64.tar.gz \
  --strip-components=3 -C /usr/local/bin \
  kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}

# etcd
tar -xf etcd-v3.5.11-linux-amd64.tar.gz
mv etcd-v3.5.11-linux-amd64/etcd /usr/local/bin/
mv etcd-v3.5.11-linux-amd64/etcdctl /usr/local/bin/
mv etcd-v3.5.11-linux-amd64/etcdutl /usr/local/bin/

同步到其他节点

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# master 节点
for NODE in master2 master3; do
  scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} $NODE:/usr/local/bin/
  scp /usr/local/bin/etcd* $NODE:/usr/local/bin/
done

# worker 节点
for NODE in worker1 worker2; do
  scp /usr/local/bin/kube{let,-proxy} $NODE:/usr/local/bin/
done

八、systemd 单元文件

8.1 kube-apiserver.service

所有 master 节点都需要,注意 --advertise-address 每个 master 不同:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-apiserver \
  --v=2 \
  --allow-privileged=true \
  --bind-address=0.0.0.0 \
  --secure-port=6443 \
  --advertise-address=192.168.139.133 \
  --service-cluster-ip-range=10.96.0.0/12,fd00:1111::/112 \
  --service-node-port-range=30000-32767 \
  --etcd-servers=https://192.168.139.133:2379,https://192.168.139.134:2379,https://192.168.139.135:2379 \
  --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \
  --etcd-certfile=/etc/etcd/ssl/etcd.pem \
  --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
  --client-ca-file=/etc/kubernetes/pki/ca.pem \
  --tls-cert-file=/etc/kubernetes/pki/apiserver.pem \
  --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \
  --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \
  --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \
  --service-account-key-file=/etc/kubernetes/pki/sa.pub \
  --service-account-signing-key-file=/etc/kubernetes/pki/sa.key \
  --service-account-issuer=https://kubernetes.default.svc.cluster.local \
  --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \
  --authorization-mode=Node,RBAC \
  --enable-bootstrap-token-auth=true \
  --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
  --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \
  --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem \
  --requestheader-allowed-names=aggregator \
  --requestheader-group-headers=X-Remote-Group \
  --requestheader-extra-headers-prefix=X-Remote-Extra- \
  --requestheader-username-headers=X-Remote-User \
  --enable-aggregator-routing=true
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

8.2 kube-controller-manager.service

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
  --v=2 \
  --bind-address=0.0.0.0 \
  --root-ca-file=/etc/kubernetes/pki/ca.pem \
  --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \
  --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \
  --service-account-private-key-file=/etc/kubernetes/pki/sa.key \
  --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \
  --leader-elect=true \
  --use-service-account-credentials=true
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

8.3 kube-scheduler.service

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-scheduler \
  --v=2 \
  --bind-address=0.0.0.0 \
  --leader-elect=true \
  --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

8.4 kubelet.service(所有 master + worker 节点)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kubelet \
  --v=2 \
  --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig \
  --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
  --config=/etc/kubernetes/kubelet-config.yaml \
  --cert-dir=/etc/kubernetes/pki \
  --hostname-override=<本节点主机名>
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

8.5 kube-proxy.service

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-proxy \
  --v=2 \
  --config=/etc/kubernetes/kube-proxy-config.yaml \
  --hostname-override=<本节点主机名>
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

九、启动顺序与验证

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# 所有 master 节点
systemctl daemon-reload
systemctl enable --now kube-apiserver kube-controller-manager kube-scheduler kubelet kube-proxy
systemctl status kube-apiserver

# 验证(master1)
kubectl get componentstatuses
# NAME                 STATUS    MESSAGE             ERROR
# controller-manager   Healthy   ok
# scheduler            Healthy   ok
# etcd-0               Healthy   {"health":"true"}

十、常见坑

  1. 证书 hostname 漏配:apiserver 报 “x509: certificate is not valid for any names”,重新签发并把 VIP、Service IP、k8s 域名都加进去
  2. bootstrap token 过期:默认 24h,过期后 worker 无法 join,kubeadm token create --print-join-command 重新生成(kubeadm 部署)
  3. etcd 启动后 apiserver 连不上:etcd 证书路径不对,配置里写 /etc/etcd/ssl/,但 cert 实际在 /etc/kubernetes/pki/etcd/,加软链
  4. kubelet 起不来:bootstrap-kubelet.kubeconfig 不存在或 token 失效
  5. kube-proxy 模式选错:默认 iptables,高并发场景(>5000 Service)改 IPVS 性能更好

十一、前置知识 / 下一步

前置

下一步

  1. etcd 集群部署(2020-09-15)—— K8s 数据层高可用
  2. API Server 高可用(2021-03-15)—— Nginx 四层 + Keepalived VIP
  3. K8s 集群插件(2021-09-15)—— CNI / CoreDNS / Metrics / Dashboard

参考资料

使用 Hugo 构建
主题 StackJimmy 设计