k8s 二进制高可用集群部署

本文档适用于 k8s 1.17+ 版本

本文章将演示 Centos7 二进制方式安装高可用 k8s 1.20+, 相对于其他版本，二进制安装方式并无太大区别，只需要区分每个组件版本的对应关系即可

生产环境中，建议使用小版本大于 5 的 kubernetes 版本，比如 1.19.5 以后的才可用于生产环境

集群安装

基本环境配置

服务器 IP 地址需改成静态 IP

虚拟IP 不要和公司内网 IP 重复，首先去 ping 一下，不通才可用。 VIP 需要和主机在同一个局域网内！

高可用 kubernetes 集群规划

主机名	IP地址	说明
k8s-master01 ~ 03	192.168.32.129 ~ 131	master 节点；3个
k8s-master-lb #有硬件负载的用硬件，云上的用云上自己的负载	192.168.32.233	keepalived 虚拟 IP
node01 ~ 02	192.168.32.132 ~ 133	worker 节点；2个

配置信息	备注
系统版本	Centos7.x
Docker 版本	19.03.x
Pod 网段	172.16.0.0/12
Service 网段	10.96.0.0/12

所有节点配置 hosts, 修改 /etc/hosts 文件如下：

[root@k8s-master01 ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.32.129 k8s-master01 192.168.32.130 k8s-master02 192.168.32.131 k8s-master03 192.168.32.132 node01 192.168.32.133 node02 192.168.32.233 k8s-master-lb

Centos 7 安装 yum 源如下；

yum install -y wget wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo #阿里yum base源 wget -P /etc/yum.repos.d/ http://mirrors.aliyun.com/repo/epel-7.repo #阿里 epel 源  yum install -y yum-utils device-mapper-persistent-data lvm2 yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo  cat > /etc/yum.repos.d/kubernetes.repo << EOF [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF  sed -i -e '/mirrors.cloud.aliyuncs.com/d' -e '/mirrors.aliyuncs.com/d' /etc/yum.repos.d/CentOS-Base.repo

必备工具安装

yum -y install wget jq psmics vim net-tools telnet yum-utils device-mapper-persistent-data lvm2 git

所有节点关闭防火墙，selinux, dnsmasq , swap

systemctl disable --now firewalld #--now 并立即生效 systemctl disable --now dnsmasq systemctl disable --now NetworkManager  setenforce 0 sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config

关闭 swap 分区

swapoff -a && sysctl -w vm.swappiness=0 #关闭 swap分区 并临时设置使用虚拟内存的权重；不建议直接关闭 swap，但是k8s集群建议关闭 swap  /etc/sysctl.conf增加一行 vm.swappiness = 0 #永久设置  sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab #注释掉 swap     内核参数vm.swappiness控制换出运行时内存的相对权重，参数值大小对如何使用swap分区有很大联系。值越大，表示越积极使用swap分区，越小表示越积极使用物理内存。默认值swappiness=60，表示内存使用率超过100-60=40%时开始使用交换分区。swappiness=0的时候表示最大限度使用物理内存，然后才是 swap空间；swappiness＝100的时候表示积极使用swap分区，并把内存上的数据及时搬运到swap空间。（网上有的说，对于3.5以后的内核和RedHat 2.6.32之后的内核，设置为0会禁止使用swap，从而引发out of memory，这种情况可以设置为1。）  需要根据服务器运行的程序类型，来设置不同的参数值。例如，对于Oracle一般设置为10；对于MySQL一般设置为1，尽可能不用swap分区。

安装 ntpdate

rpm -ivh http://mirrors.wlnmp.com/centos/wlnmp-release-centos.noarch.rpm yum -y install ntpdate

所有节点同步时间，时间同步配置如下

ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime #描述本机时间 echo 'Asia/Shanghai' > /etc/timezone #描述本机所属的时区 ntpdate time2.aliyun.com  加入到 crontab [root@k8s-master01 yum.repos.d]# crontab -l */5 * * * * /usr/sbin/ntpdate time2.aliyun.com

所有节点配置 limit

 ulimit -SHn 65535 #设置当前shell,所有用户的最大文件打开数位 65535个，立即生效 vim /etc/security/limits.conf #永久生效，新开shell，立即生效，默认以这个配置文件为准，如果不去手动设置 # 末尾添加如下内容 [root@k8s-master01 yum.repos.d]# tail -6 /etc/security/limits.conf * soft nofile 655360 * hard nofile 131072 * soft nproc 655350 * hard nproc 655350 * soft memlock unlimited * hard memlock unlimited [root@k8s-master01 yum.repos.d]#  ulimit 是一种 Linux 系统的内键功能，它具有一套参数集，用于为由它生成的 shell进程及其子进程的资源使用设置限制。 ulimit的设定值是 per-process 的，也就是说，每个进程有自己的limits值。 使用ulimit进行修改，是 立即生效 的。 ulimit只影响shell进程及其子进程，用户登出后失效。    #nofile 最大文件打开数 #soft 可以超过，会报警 #hard 硬性规定，不能超过 #memlock - 最大锁定内存地址空间 #noproc - 进程的最大数目 #unlimited 无限制的

Master01 节点免密钥登陆其他节点，安装过程中生成配置文件和证书（kubeadm不需要手动生成证书)均在 Master01 上操作，集群管理也在 Master01 上操作，阿里云或者 AWS 上需要单独一台 kubectl 服务器。密钥配置如下；

ssh-keygen -t rsa for i in k8s-master01 k8s-master02 k8s-master03 node01 node02;do ssh-copy-id -i /root/.ssh/id_rsa.pub $i;done

下载安装所有的源码文件（master01上操作就行了）

git clone https://gitee.com/luoluo160717/k8s-ha-install.git

所有节点升级系统并重启，此处升级没有升级内核，下面会单独升级内核

 yum update -y --exclude=kernel* && reboot #CentOs7 需要升级

内核配置

Centos7 需要升级内核至4.18+，本次升级的版本为4.19

在 master01 节点下载内核

cd /root wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm  for i in k8s-master01 k8s-master02 k8s-master03 node01 node02;do scp kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm $i:/root/;done #分发到其他节点

所有节点升级内核

yum localinstall -y kernel-ml*

所有节点更改内核启动顺序

grub2-set-default 0 && grub2-mkconfig -o /etc/grub2.cfg grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"

检查默认内核是不是4.19

[root@k8s-master01 ~]# grubby --default-kernel /boot/vmlinuz-4.19.12-1.el7.elrepo.x86_64

所有节点重启，然后检查内核是不是 4.19

[root@k8s-master01 ~]# uname -a Linux k8s-master01 4.19.12-1.el7.elrepo.x86_64 #1 SMP Fri Dec 21 11:06:36 EST 2018 x86_64 x86_64 x86_64 GNU/Linux

所有节点安装 ipvsadm

yum install ipvsadm ipset sysstat conntrack libseccomp -y

所有节点配置 ipvs 模块，在内核 4.19+ 版本 nf_conntrack_ipv4 已经改为 nf_conntrack, 4.18 以下使用 nf_conntrack_ipv4 即可

vim /etc/modules-load.d/ipvs.conf #默认不存在 [root@k8s-master01 ~]# cat /etc/modules-load.d/ipvs.conf #加入如下配置 ip_vs ip_vs_lc ip_vs_wlc ip_vs_rr ip_vs_wrr ip_vs_lblc ip_vs_lblcr ip_vs_dh ip_vs_sh ip_vs_fo ip_vs_nq ip_vs_sed ip_vs_ftp ip_vs_sh nf_conntrack ip_tables ip_set xt_set ipt_set ipt_rpfilter ipt_REJECT ipip     然后执行 systemctl enable --now systemd-modules-load.service 即可

开启一些k8s集群中必须的内核参数，所有节点配置k8s内核

[root@k8s-master01 ~]# cat <<EOF > /etc/sysctl.d/k8s.conf net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 fs.may_detach_mounts = 1 net.ipv4.conf.all.route_localnet = 1 vm.overcommit_memory=1 vm.panic_on_oom=0 fs.inotify.max_user_watches=89100 fs.file-max=52706963 fs.nr_open=52706963 net.netfilter.nf_conntrack_max=2310720 net.ipv4.tcp_keepalive_time = 600 net.ipv4.tcp_keepalive_probes = 3 net.ipv4.tcp_keepalive_intvl =15 net.ipv4.tcp_max_tw_buckets = 36000 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_max_orphans = 327680 net.ipv4.tcp_orphan_retries = 3 net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_max_syn_backlog = 16384 net.ipv4.ip_conntrack_max = 65536 net.ipv4.tcp_max_syn_backlog = 16384 net.ipv4.tcp_timestamps = 0 net.core.somaxconn = 16384 EOF   执行 sysctl --system #加载所有的配置文件

所有节点配置完内核后，重启服务器，保证重启后内核加载

reboot lsmod | grep --color=auto -e ip_vs -e nf_conntrack

基本组件安装

docker-ce, kubernetes 各组件等

所有节点安装 docker-ce 19.03

yum install docker-ce-19.03.* -y

温馨提示；

由于新版 kubelet 建议使用 systemd , 所以可以把 docker 的 CgroupDriver 改成 systemd

mkdir /etc/docker cat > /etc/docker/daemon.json <<EOF { "registry-mirrors": ["https://6h6ezoe5.mirror.aliyuncs.com"], "exec-opts": ["native.cgroupdriver=systemd"] } EOF

所有节点设置开机自启动 docker

systemctl daemon-reload && systemctl enable --now docker

k8s 及 etcd 安装

Master01 下载 kubernetes 二进制 tar.gz 压缩包

GitHub - kubernetes/kubernetes: Production-Grade Container Scheduling and Management

选择对应版本的CHANGELOG-1.X.md文件，点击打开

点击 Server Binaries

右键复制链接或者鼠标点击下载

wget https://dl.k8s.io/v1.20.14/kubernetes-server-linux-amd64.tar.gz #复制的链接地址  如果网络原因上不去 github ， 可以将其中dl.k8s.io修改为 storage.googleapis.com/kubernetes-release/release  新的下载地址：wget https://storage.googleapis.com/kubernetes-release/release/v1.20.14/kubernetes-server-linux-amd64.tar.gz

Releases · etcd-io/etcd (github.com) #

etcd 二进制包 地址：https://github.com/etcd-io/etcd/releases/download/v3.4.13/etcd-v3.4.13-linux-amd64.tar.gz   下载不下来的可以去网盘下载 链接：https://pan.baidu.com/s/16k1WGvhRsdu2iro-GbkNBg 提取码：3941 --来自百度网盘超级会员V1的分享

解压包

[root@k8s-master01 ~]# tar -xf kubernetes-server-linux-amd64.tar.gz --strip-components 3 -C /usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} #解压指定内容到 /usr/local/bin/ 下，    [root@k8s-master01 ~]# tar -xf etcd-v3.4.13-linux-amd64.tar.gz --strip-components 1 -C /usr/local/bin/ etcd-v3.4.13-linux-amd64/etcd{,ctl}

版本查看

[root@k8s-master01 ~]# kubelet --version Kubernetes v1.20.14 [root@k8s-master01 ~]# etcdctl version etcdctl version: 3.4.13 API version: 3.4 [root@k8s-master01 ~]#

将组件发送到其他节点

[root@k8s-master01 ~]# masternode='k8s-master02 k8s-master03' [root@k8s-master01 ~]# worknode='node01 node02'  [root@k8s-master01 ~]# for i in $masternode;do scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} $i:/usr/local/bin/; scp /usr/local/bin/etcd* $i:/usr/local/bin/;done   [root@k8s-master01 ~]# for i in $worknode;do scp /usr/local/bin/kube{let,-proxy} $i:/usr/local/bin/; done

所有节点创建 /opt/cni/bin 目录

mkdir -p /opt/cni/bin

master01 上操作，切换到1.20.x 分支（其他版本可以切换到其他分支)

[root@k8s-master01 ~]# cd /root/k8s-ha-install/ You have new mail in /var/spool/mail/root [root@k8s-master01 k8s-ha-install]# git checkout manual-installation-v1.20.x Branch manual-installation-v1.20.x set up to track remote branch manual-installation-v1.20.x from origin. Switched to a new branch 'manual-installation-v1.20.x' [root@k8s-master01 k8s-ha-install]#

生成证书

单向认证双向认证

何为SSL/TLS单向认证，双向认证？ 单向认证指的是只有一个对象校验对端的证书合法性。通常都是client来校验服务器的合法性。那么client需要一个ca.crt,服务器需要server.crt,server.key 双向认证指的是相互校验，服务器需要校验每个client,client也需要校验服务器。 server 需要 server.key 、server.crt 、ca.crt client 需要 client.key 、client.crt 、ca.crt

kubernetesV1.8版本后建议开启TLS双向认证及RBAC授权管理，以加强集群的安全管理。界内流行的开启TLS方法为基于一个“公钥基础设施（public key infrastructure，缩写为 PKI）”，使用了内部托管的认证中心（CA），常见PKI工具有CFSSL,OPENSSL等

CFSSL是CloudFlare开源的一款PKI/TLS工具。 CFSSL 包含一个命令行工具和一个用于签名，验证并且捆绑TLS证书的 HTTP API 服务。使用Go语言编写。

CFSSL包括：

一组用于生成自定义 TLS PKI 的工具
cfssl程序，是CFSSL的命令行工具
multirootca程序是可以使用多个签名密钥的证书颁发机构服务器
mkbundle程序用于构建证书池
cfssljson程序，从cfssl和multirootca程序获取JSON输出，并将证书，密钥，CSR和bundle写入磁盘

PKI借助数字证书和公钥加密技术提供可信任的网络身份。通常，证书就是一个包含如下身份信息的文件：

证书所有组织的信息
公钥
证书颁发组织的信息
证书颁发组织授予的权限，如证书有效期、适用的主机名、用途等
使用证书颁发组织私钥创建的数字签名

安装cfssl

这里我们只用到cfssl工具和cfssljson工具：

Master01 下载生成证书工具

[root@k8s-master01 ~]# wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -O /usr/local/bin/cfssl  [root@k8s-master01 ~]# wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -O /usr/local/bin/cfssljson  [root@k8s-master01 ~]# chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson    网络问题下载不下来的可以去网盘下载 链接：https://pan.baidu.com/s/1VcGd_FQE5GoKsaU3Jx79QA 提取码：3941 --来自百度网盘超级会员V1的分享

cfssl工具，子命令介绍：

bundle: 创建包含客户端证书的证书包
genkey: 生成一个key(私钥)和CSR(证书签名请求)
scan: 扫描主机问题
revoke: 吊销证书
certinfo: 输出给定证书的证书信息，跟cfssl-certinfo 工具作用一样
gencrl: 生成新的证书吊销列表
selfsign: 生成一个新的自签名密钥和签名证书
print-defaults: 打印默认配置，这个默认配置可以用作模板
serve: 启动一个HTTP API服务
gencert: 生成新的key(密钥)和签名证书
- -ca：指明ca的证书
- -ca-key：指明ca的私钥文件
- -config：指明请求证书的json文件
- -profile：与-config中的profile对应，是指根据config中的profile段来生成证书的相关信息
ocspdump
ocspsign
info: 获取有关远程签名者的信息
sign: 签名一个客户端证书，通过给定的CA和CA密钥，和主机名
ocsprefresh
ocspserve

cfssl常用命令：

cfssl gencert -initca ca-csr.json | cfssljson -bare ca ## 初始化ca
cfssl gencert -initca -ca-key key.pem ca-csr.json | cfssljson -bare ca ## 使用现有私钥, 重新生成
cfssl certinfo -cert ca.pem
cfssl certinfo -csr ca.csr

所有Master 节点创建 etcd 证书目录

[root@k8s-master01 ~]# mkdir /etc/etcd/ssl -p

所有节点创建 kubernetes 相关目录

[root@k8s-master01 ~]# mkdir -p /etc/kubernetes/pki

Master01 节点生成 etcd 证书

生成证书的 CSR 文件：证书签名请求文件，配置了一些域名，公司，单位

cd /root/k8s-ha-install/pki  ##cfssl print-defaults config > config.json # 默认证书生产策略配置模板 ##cfssl print-defaults csr > csr.json #默认csr请求模板     初始化生成 etcd CA 根证书和 etcd CA 证书的 key  [root@k8s-master01 pki]# cfssl gencert -initca etcd-csr.json | cfssljson -bare /etc/etcd/ssl/etcd-ca  [root@k8s-master01 pki]# ls /etc/etcd/ssl/ etcd-ca.csr etcd-ca-key.pem etcd-ca.pem  生成了 etcd ca的证书 etcd-ca.pem 生产了 etcd ca的私钥 etcd-ca-key.pem 生成了 etcd ca的请求文件 etcd-ca.csr      生成客户端证书   [root@k8s-master01 pki]# cfssl gencert -ca=/etc/etcd/ssl/etcd-ca.pem -ca-key=/etc/etcd/ssl/etcd-ca-key.pem -config=ca-config.json -hostname=127.0.0.1,k8s-master01,k8s-master02,k8s-master03,192.168.32.129,192.168.32.130,192.168.32.131,192.168.32.144,192.168.32.145 -profile=kubernetes etcd-csr.json | cfssljson -bare /etc/etcd/ssl/etcd    192.168.32.144 192.168.32.145 #预留地址 可以多写几个

将证书复制到其他 etcd 节点，也就是本实验中的,master02,master03节点

上边已经生成过变量，请注意  [root@k8s-master01 ~]# for i in $masternode;do ssh $i "mkdir -p /etc/etcd/ssl";for j in etcd-ca-key.pem etcd-ca.pem etcd-key.pem etcd.pem;do scp /etc/etcd/ssl/${j} $i:/etc/etcd/ssl/${j};done;done

Master01节点生成 kubernetes 证书

初始化根证书 ,用于k8s 集群 [root@k8s-master01 pki]# cfssl gencert -initca ca-csr.json | cfssljson -bare /etc/kubernetes/pki/ca           这个证书目前专属于 apiserver，加了一个 k8s-master* 域名以便内部私有 DNS 解析使用(可删除)；至于很多人问过 kubernetes 这几个能不能删掉，答案是不可以的；因为当集群创建好后，default namespace 下会创建一个叫 kubenretes 的 svc，有一些组件会直接连接这个 svc 来跟 api 通讯的，证书如果不包含可能会出现无法连接的情况；其他几个 kubernetes 开头的域名作用相同 hosts包含的是授权范围，不在此范围的的节点或者服务使用此证书就会报证书不匹配错误。 10.96.0.1是指kube-apiserver 指定的 service-cluster-ip-range 网段的第一个IP。 #-hostname 配置区域，即一个证书的网站可以是*.youku.com也是可以是*.google.com  [root@k8s-master01 pki]# cfssl gencert -ca=/etc/kubernetes/pki/ca.pem -ca-key=/etc/kubernetes/pki/ca-key.pem -config=ca-config.json -hostname=10.96.0.1,192.168.32.233,127.0.0.1,k8s-master*,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,192.168.32.129,192.168.32.130,192.168.32.131 -profile=kubernetes apiserver-csr.json | cfssljson -bare /etc/kubernetes/pki/apiserver         初始化生成apiserver的聚合证书  [root@k8s-master01 pki]# cfssl gencert -initca front-proxy-ca-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-ca    [root@k8s-master01 pki]# cfssl gencert -ca=/etc/kubernetes/pki/front-proxy-ca.pem -ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem -config=ca-config.json -profile=kubernetes front-proxy-client-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-client              生成controller-manage的证书  [root@k8s-master01 pki]# cfssl gencert \ -ca=/etc/kubernetes/pki/ca.pem \ -ca-key=/etc/kubernetes/pki/ca-key.pem \ -config=ca-config.json \ -profile=kubernetes \ manager-csr.json | cfssljson -bare /etc/kubernetes/pki/controller-manager # 注意，如果不是高可用集群，192.168.32.233:8443改为master01的地址，8443改为apiserver的端口，默认是6443 # set-cluster：设置一个集群项,192.168.32.233是VIP kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/pki/ca.pem \ --embed-certs=true \ --server=https://192.168.32.233:8443 \ --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig   # 设置一个环境项，一个上下文 [root@k8s-master01 pki]# kubectl config set-context system:kube-controller-manager@kubernetes \ --cluster=kubernetes \ --user=system:kube-controller-manager \ --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig        # set-credentials 设置一个用户项 [root@k8s-master01 pki]# kubectl config set-credentials system:kube-controller-manager \ --client-certificate=/etc/kubernetes/pki/controller-manager.pem \ --client-key=/etc/kubernetes/pki/controller-manager-key.pem \ --embed-certs=true \ --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig # 使用某个环境当做默认环境 [root@k8s-master01 pki]# kubectl config use-context system:kube-controller-manager@kubernetes \ --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig 生成 scheduler 证书 [root@k8s-master01 pki]# cfssl gencert \ -ca=/etc/kubernetes/pki/ca.pem \ -ca-key=/etc/kubernetes/pki/ca-key.pem \ -config=ca-config.json \ -profile=kubernetes \ scheduler-csr.json | cfssljson -bare /etc/kubernetes/pki/scheduler   # 注意，如果不是高可用集群，192.168.32.233:8443改为master01的地址，8443改为apiserver的端口，默认是6443   [root@k8s-master01 pki]# kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/pki/ca.pem \ --embed-certs=true \ --server=https://192.168.32.233:8443 \ --kubeconfig=/etc/kubernetes/scheduler.kubeconfig kubectl config set-credentials system:kube-scheduler \ --client-certificate=/etc/kubernetes/pki/scheduler.pem \ --client-key=/etc/kubernetes/pki/scheduler-key.pem \ --embed-certs=true \ --kubeconfig=/etc/kubernetes/scheduler.kubeconfig kubectl config set-context system:kube-scheduler@kubernetes \ --cluster=kubernetes \ --user=system:kube-scheduler \ --kubeconfig=/etc/kubernetes/scheduler.kubeconfig kubectl config use-context system:kube-scheduler@kubernetes \ --kubeconfig=/etc/kubernetes/scheduler.kubeconfig cfssl gencert \ -ca=/etc/kubernetes/pki/ca.pem \ -ca-key=/etc/kubernetes/pki/ca-key.pem \ -config=ca-config.json \ -profile=kubernetes \ admin-csr.json | cfssljson -bare /etc/kubernetes/pki/admin # 注意，如果不是高可用集群，192.168.32.233:8443改为master01的地址，8443改为apiserver的端口，默认是6443 kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.pem --embed-certs=true --server=https://192.168.32.233:8443 --kubeconfig=/etc/kubernetes/admin.kubeconfig    kubectl config set-credentials kubernetes-admin --client-certificate=/etc/kubernetes/pki/admin.pem --client-key=/etc/kubernetes/pki/admin-key.pem --embed-certs=true --kubeconfig=/etc/kubernetes/admin.kubeconfig     kubectl config set-context kubernetes-admin@kubernetes --cluster=kubernetes --user=kubernetes-admin --kubeconfig=/etc/kubernetes/admin.kubeconfig    kubectl config use-context kubernetes-admin@kubernetes --kubeconfig=/etc/kubernetes/admin.kubeconfig

创建ServiceAccount Key

[root@k8s-master01 pki]# openssl genrsa -out /etc/kubernetes/pki/sa.key 2048   [root@k8s-master01 pki]# openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/sa.pub

发送证书至其他节点

for NODE in k8s-master02 k8s-master03; do for FILE in $(ls /etc/kubernetes/pki | grep -v etcd); do scp /etc/kubernetes/pki/${FILE} $NODE:/etc/kubernetes/pki/${FILE}; done; for FILE in admin.kubeconfig controller-manager.kubeconfig scheduler.kubeconfig; do scp /etc/kubernetes/${FILE} $NODE:/etc/kubernetes/${FILE}; done; done;

查看证书文件

[root@k8s-master01 pki]# ls /etc/kubernetes/pki/ admin.csr apiserver.csr ca.csr controller-manager.csr front-proxy-ca.csr front-proxy-client.csr sa.key scheduler-key.pem admin-key.pem apiserver-key.pem ca-key.pem controller-manager-key.pem front-proxy-ca-key.pem front-proxy-client-key.pem sa.pub scheduler.pem admin.pem apiserver.pem ca.pem controller-manager.pem front-proxy-ca.pem front-proxy-client.pem scheduler.csr  [root@k8s-master01 pki]# ls /etc/kubernetes/pki/ |wc -l 23

Kubernetes系统组件配置

Etcd配置（所有Master节点）

etcd配置大致相同，注意修改每个Master节点的etcd配置的主机名和IP地址

Master01 节点

[root@k8s-master01 ~]# vim /etc/etcd/etcd.config.yml  name: 'k8s-master01' data-dir: /var/lib/etcd wal-dir: /var/lib/etcd/wal snapshot-count: 5000 heartbeat-interval: 100 election-timeout: 1000 quota-backend-bytes: 0 listen-peer-urls: 'https://192.168.32.129:2380' listen-client-urls: 'https://192.168.32.129:2379,http://127.0.0.1:2379' max-snapshots: 3 max-wals: 5 cors: initial-advertise-peer-urls: 'https://192.168.32.129:2380' advertise-client-urls: 'https://192.168.32.129:2379' discovery: discovery-fallback: 'proxy' discovery-proxy: discovery-srv: initial-cluster: 'k8s-master01=https://192.168.32.129:2380,k8s-master02=https://192.168.32.130:2380,k8s-master03=https://192.168.32.131:2380' initial-cluster-token: 'etcd-k8s-cluster' initial-cluster-state: 'new' strict-reconfig-check: false enable-v2: true enable-pprof: true proxy: 'off' proxy-failure-wait: 5000 proxy-refresh-interval: 30000 proxy-dial-timeout: 1000 proxy-write-timeout: 5000 proxy-read-timeout: 0 client-transport-security: cert-file: '/etc/kubernetes/pki/etcd/etcd.pem' key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem' client-cert-auth: true trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem' auto-tls: true peer-transport-security: cert-file: '/etc/kubernetes/pki/etcd/etcd.pem' key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem' peer-client-cert-auth: true trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem' auto-tls: true debug: false log-package-levels: log-outputs: [default] force-new-cluster: false

Master02 节点

[root@k8s-master02 ~]# vim /etc/etcd/etcd.config.yml name: 'k8s-master02' data-dir: /var/lib/etcd wal-dir: /var/lib/etcd/wal snapshot-count: 5000 heartbeat-interval: 100 election-timeout: 1000 quota-backend-bytes: 0 listen-peer-urls: 'https://192.168.32.130:2380' listen-client-urls: 'https://192.168.32.130:2379,http://127.0.0.1:2379' max-snapshots: 3 max-wals: 5 cors: initial-advertise-peer-urls: 'https://192.168.32.130:2380' advertise-client-urls: 'https://192.168.32.130:2379' discovery: discovery-fallback: 'proxy' discovery-proxy: discovery-srv: initial-cluster: 'k8s-master01=https://192.168.32.129:2380,k8s-master02=https://192.168.32.130:2380,k8s-master03=https://192.168.32.131:2380' initial-cluster-token: 'etcd-k8s-cluster' initial-cluster-state: 'new' strict-reconfig-check: false enable-v2: true enable-pprof: true proxy: 'off' proxy-failure-wait: 5000 proxy-refresh-interval: 30000 proxy-dial-timeout: 1000 proxy-write-timeout: 5000 proxy-read-timeout: 0 client-transport-security: cert-file: '/etc/kubernetes/pki/etcd/etcd.pem' key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem' client-cert-auth: true trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem' auto-tls: true peer-transport-security: cert-file: '/etc/kubernetes/pki/etcd/etcd.pem' key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem' peer-client-cert-auth: true trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem' auto-tls: true debug: false log-package-levels: log-outputs: [default] force-new-cluster: false

Master03 节点

[root@k8s-master03 ~]# vim /etc/etcd/etcd.config.yml name: 'k8s-master03' data-dir: /var/lib/etcd wal-dir: /var/lib/etcd/wal snapshot-count: 5000 heartbeat-interval: 100 election-timeout: 1000 quota-backend-bytes: 0 listen-peer-urls: 'https://192.168.32.131:2380' listen-client-urls: 'https://192.168.32.131:2379,http://127.0.0.1:2379' max-snapshots: 3 max-wals: 5 cors: initial-advertise-peer-urls: 'https://192.168.32.131:2380' advertise-client-urls: 'https://192.168.32.131:2379' discovery: discovery-fallback: 'proxy' discovery-proxy: discovery-srv: initial-cluster: 'k8s-master01=https://192.168.32.129:2380,k8s-master02=https://192.168.32.130:2380,k8s-master03=https://192.168.32.131:2380' initial-cluster-token: 'etcd-k8s-cluster' initial-cluster-state: 'new' strict-reconfig-check: false enable-v2: true enable-pprof: true proxy: 'off' proxy-failure-wait: 5000 proxy-refresh-interval: 30000 proxy-dial-timeout: 1000 proxy-write-timeout: 5000 proxy-read-timeout: 0 client-transport-security: cert-file: '/etc/kubernetes/pki/etcd/etcd.pem' key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem' client-cert-auth: true trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem' auto-tls: true peer-transport-security: cert-file: '/etc/kubernetes/pki/etcd/etcd.pem' key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem' peer-client-cert-auth: true trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem' auto-tls: true debug: false log-package-levels: log-outputs: [default] force-new-cluster: false

创建Service

创建etcd service并启动（所有Master节点）

vim /usr/lib/systemd/system/etcd.service [Unit] Description=Etcd Service Documentation=https://coreos.com/etcd/docs/latest/ After=network.target  [Service] Type=notify ExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/etcd.config.yml Restart=on-failure RestartSec=10 LimitNOFILE=65536  [Install] WantedBy=multi-user.target Alias=etcd3.service

创建etcd的证书目录（所有Master节点）

  mkdir /etc/kubernetes/pki/etcd ln -s /etc/etcd/ssl/* /etc/kubernetes/pki/etcd/ systemctl daemon-reload systemctl enable --now etcd

查看集群状态（任意master）

[root@k8s-master01 ~]# ETCDCTL_API=3 [root@k8s-master01 ~]# etcdctl --endpoints="192.168.32.129:2379,192.168.32.130:2379,192.168.32.131:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem endpoint status --write-out=table +---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | 192.168.32.129:2379 | c391821cb3f30b1a | 3.4.13 | 25 kB | false | false | 2315 | 10 | 10 | | | 192.168.32.130:2379 | d65c253897bed4d4 | 3.4.13 | 20 kB | true | false | 2315 | 10 | 10 | | | 192.168.32.131:2379 | f6e8226e0b0ed00e | 3.4.13 | 20 kB | false | false | 2315 | 10 | 10 | | +---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ [root@k8s-master01 ~]#

高可用配置

高可用配置（注意：如果不是高可用集群，haproxy和keepalived无需安装）如果在云上安装也无需执行此章节的步骤，可以直接使用云上的服务

安装keepalived和haproxy（所有Master节点）

yum install keepalived haproxy -y

所有Master节点配置HAProxy，配置都一样

vim /etc/haproxy/haproxy.cfg global maxconn 2000 ulimit-n 16384 log 127.0.0.1 local0 err stats timeout 30s  defaults log global mode http option httplog timeout connect 5000 timeout client 50000 timeout server 50000 timeout http-request 15s timeout http-keep-alive 15s  frontend k8s-master bind 0.0.0.0:8443 bind 127.0.0.1:8443 mode tcp option tcplog tcp-request inspect-delay 5s default_backend k8s-master  backend k8s-master mode tcp option tcplog option tcp-check balance roundrobin default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100 server k8s-master01 192.168.32.129:6443 check server k8s-master02 192.168.32.130:6443 check server k8s-master03 192.168.32.131:6443 check

配置KeepAlived（Master节点）

注意每个节点的IP和网卡（interface参数）

Master01

vim /etc/keepalived/keepalived.conf  ! Configuration File for keepalived global_defs { router_id LVS_DEVEL } vrrp_script chk_apiserver { script "/etc/keepalived/check_apiserver.sh" interval 5 weight -5 fall 2 rise 1 } vrrp_instance VI_1 { state MASTER interface ens32 mcast_src_ip 192.168.32.129 virtual_router_id 51 priority 101 nopreempt advert_int 2 authentication { auth_type PASS auth_pass K8SHA_KA_AUTH } virtual_ipaddress { 192.168.32.233 } track_script { chk_apiserver } }

Master02

vim /etc/keepalived/keepalived.conf  ! Configuration File for keepalived global_defs { router_id LVS_DEVEL } vrrp_script chk_apiserver { script "/etc/keepalived/check_apiserver.sh" interval 5 weight -5 fall 2 rise 1 } vrrp_instance VI_1 { state BACKUP interface ens32 mcast_src_ip 192.168.32.130 virtual_router_id 51 priority 100 nopreempt advert_int 2 authentication { auth_type PASS auth_pass K8SHA_KA_AUTH } virtual_ipaddress { 192.168.32.233 } track_script { chk_apiserver } }

Master03

vim /etc/keepalived/keepalived.conf  ! Configuration File for keepalived global_defs { router_id LVS_DEVEL } vrrp_script chk_apiserver { script "/etc/keepalived/check_apiserver.sh" interval 5 weight -5 fall 2 rise 1 } vrrp_instance VI_1 { state BACKUP interface ens32 mcast_src_ip 192.168.32.131 virtual_router_id 51 priority 100 nopreempt advert_int 2 authentication { auth_type PASS auth_pass K8SHA_KA_AUTH } virtual_ipaddress { 192.168.32.233 } track_script { chk_apiserver } }

健康检查脚本（所有master节点）

cat > /etc/keepalived/check_apiserver.sh << EFO #!/bin/bash  err=0 for k in $(seq 1 3) do check_code=$(pgrep haproxy) if [[ $check_code == "" ]]; then err=$(expr $err + 1) sleep 1 continue else err=0 break fi done  if [[ $err != "0" ]]; then echo "systemctl stop keepalived" /usr/bin/systemctl stop keepalived exit 1 else exit 0 fi EFO   # 授权 chmod +x /etc/keepalived/check_apiserver.sh

节点启动haproxy和keepalived（所有master节点）

systemctl daemon-reload systemctl enable --now haproxy systemctl enable --now keepalived

VIP测试（master01）

重要：如果安装了keepalived和haproxy，需要测试keepalived是否是正常的

# 看到有VIP绑定到ens33网卡上了 [root@k8s-master01 pki]# ip addr show ens32 2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:ce:bd:c6 brd ff:ff:ff:ff:ff:ff inet 192.168.32.129/24 brd 192.168.32.255 scope global dynamic ens32 valid_lft 1164sec preferred_lft 1164sec inet 192.168.32.233/32 scope global ens32 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:fece:bdc6/64 scope link valid_lft forever preferred_lft forever [root@k8s-master01 pki]#   # 任意节点，检查haproxy telnet 192.168.32.233 8443   如果ping不通且telnet没有出现 "]"，则认为VIP不可以，不可在继续往下执行，需要排查keepalived的问题，比如防火墙和selinux，haproxy和keepalived的状态，监听端口等 所有节点查看防火墙状态必须为disable和inactive：systemctl status firewalld 所有节点查看selinux状态，必须为disable：getenforce master节点查看haproxy和keepalived状态：systemctl status keepalived haproxy master节点查看监听端口：netstat -lntp

Kubernetes组件配置

Apiserver

所有Master节点创建kube-apiserver service，# 注意，如果不是高可用集群，192.168.32.233改为master01的地址

注意本文档使用的k8s service网段为10.96.0.0/12，该网段不能和宿主机的网段、Pod网段的重复，请按需修改

Master01配置

[root@k8s-master01 ~]# vim /usr/lib/systemd/system/kube-apiserver.service  [Unit] Description=Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes After=network.target  [Service] ExecStart=/usr/local/bin/kube-apiserver \ --v=2 \ --logtostderr=true \ --allow-privileged=true \ --bind-address=0.0.0.0 \ --secure-port=6443 \ --insecure-port=0 \ --advertise-address=192.168.32.129 \ # 主机IP 记得把注释删了 --service-cluster-ip-range=10.96.0.0/12 \ # service网段 --service-node-port-range=30000-32767 \ --etcd-servers=https://192.168.32.129:2379,https://192.168.32.130:2379,https://192.168.32.131:2379 \ --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \ --etcd-certfile=/etc/etcd/ssl/etcd.pem \ --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \ --client-ca-file=/etc/kubernetes/pki/ca.pem \ --tls-cert-file=/etc/kubernetes/pki/apiserver.pem \ --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \ --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \ --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \ --service-account-key-file=/etc/kubernetes/pki/sa.pub \ --service-account-signing-key-file=/etc/kubernetes/pki/sa.key \ --service-account-issuer=https://kubernetes.default.svc.cluster.local \ --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \ --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \ --authorization-mode=Node,RBAC \ --enable-bootstrap-token-auth=true \ --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \ --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \ --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem \ --requestheader-allowed-names=aggregator \ --requestheader-group-headers=X-Remote-Group \ --requestheader-extra-headers-prefix=X-Remote-Extra- \ --requestheader-username-headers=X-Remote-User # --token-auth-file=/etc/kubernetes/token.csv  Restart=on-failure RestartSec=10s LimitNOFILE=65535  [Install] WantedBy=multi-user.target

Master02

[root@k8s-master02 ~]# vim /usr/lib/systemd/system/kube-apiserver.service  [Unit] Description=Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes After=network.target  [Service] ExecStart=/usr/local/bin/kube-apiserver \ --v=2 \ --logtostderr=true \ --allow-privileged=true \ --bind-address=0.0.0.0 \ --secure-port=6443 \ --insecure-port=0 \ --advertise-address=192.168.32.130 \ # 主机IP --service-cluster-ip-range=10.96.0.0/12 \ # servicer IP --service-node-port-range=30000-32767 \ --etcd-servers=https://192.168.32.129:2379,https://192.168.32.130:2379,https://192.168.32.131:2379 \ --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \ --etcd-certfile=/etc/etcd/ssl/etcd.pem \ --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \ --client-ca-file=/etc/kubernetes/pki/ca.pem \ --tls-cert-file=/etc/kubernetes/pki/apiserver.pem \ --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \ --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \ --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \ --service-account-key-file=/etc/kubernetes/pki/sa.pub \ --service-account-signing-key-file=/etc/kubernetes/pki/sa.key \ --service-account-issuer=https://kubernetes.default.svc.cluster.local \ --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \ --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \ --authorization-mode=Node,RBAC \ --enable-bootstrap-token-auth=true \ --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \ --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \ --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem \ --requestheader-allowed-names=aggregator \ --requestheader-group-headers=X-Remote-Group \ --requestheader-extra-headers-prefix=X-Remote-Extra- \ --requestheader-username-headers=X-Remote-User # --token-auth-file=/etc/kubernetes/token.csv  Restart=on-failure RestartSec=10s LimitNOFILE=65535  [Install] WantedBy=multi-user.target

Master03

[root@k8s-master03 ~]# vim /usr/lib/systemd/system/kube-apiserver.service  [Unit] Description=Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes After=network.target  [Service] ExecStart=/usr/local/bin/kube-apiserver \ --v=2 \ --logtostderr=true \ --allow-privileged=true \ --bind-address=0.0.0.0 \ --secure-port=6443 \ --insecure-port=0 \ --advertise-address=192.168.32.131 \ # 主机IP --service-cluster-ip-range=10.96.0.0/12 \ # servicer IP --service-node-port-range=30000-32767 \ --etcd-servers=https://192.168.32.129:2379,https://192.168.32.130:2379,https://192.168.32.131:2379 \ --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \ --etcd-certfile=/etc/etcd/ssl/etcd.pem \ --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \ --client-ca-file=/etc/kubernetes/pki/ca.pem \ --tls-cert-file=/etc/kubernetes/pki/apiserver.pem \ --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \ --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \ --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \ --service-account-key-file=/etc/kubernetes/pki/sa.pub \ --service-account-signing-key-file=/etc/kubernetes/pki/sa.key \ --service-account-issuer=https://kubernetes.default.svc.cluster.local \ --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \ --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \ --authorization-mode=Node,RBAC \ --enable-bootstrap-token-auth=true \ --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \ --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \ --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem \ --requestheader-allowed-names=aggregator \ --requestheader-group-headers=X-Remote-Group \ --requestheader-extra-headers-prefix=X-Remote-Extra- \ --requestheader-username-headers=X-Remote-User # --token-auth-file=/etc/kubernetes/token.csv  Restart=on-failure RestartSec=10s LimitNOFILE=65535  [Install] WantedBy=multi-user.target

启动apiserver（所有Master节点）

systemctl daemon-reload && systemctl enable --now kube-apiserver   # 检测kube-server状态 systemctl status kube-apiserver

配置kube-controller-manager service （所有Master节点）

注意本文档使用的k8s Pod网段为172.16.0.0/12，该网段不能和宿主机的网段、k8s Service网段的重复，请按需修改

三个Master 节点配置文件内容相同  vim /usr/lib/systemd/system/kube-controller-manager.service [Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/kubernetes/kubernetes After=network.target  [Service] ExecStart=/usr/local/bin/kube-controller-manager \ --v=2 \ --logtostderr=true \ --address=127.0.0.1 \ --root-ca-file=/etc/kubernetes/pki/ca.pem \ --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \ --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \ --service-account-private-key-file=/etc/kubernetes/pki/sa.key \ --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \ --leader-elect=true \ --use-service-account-credentials=true \ --node-monitor-grace-period=40s \ --node-monitor-period=5s \ --pod-eviction-timeout=2m0s \ --controllers=*,bootstrapsigner,tokencleaner \ --allocate-node-cidrs=true \ --cluster-cidr=172.16.0.0/12 \ --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \ --node-cidr-mask-size=24 Restart=always RestartSec=10s  [Install] WantedBy=multi-user.target

启动（所有matser节点）

systemctl daemon-reload && systemctl enable --now kube-controller-manager

查看状态（所有matser节点）

systemctl status kube-controller-manager

配置kube-scheduler service（所有Master节点）

所有master节点配置文件相同

vim /usr/lib/systemd/system/kube-scheduler.service  [Unit] Description=Kubernetes Scheduler Documentation=https://github.com/kubernetes/kubernetes After=network.target  [Service] ExecStart=/usr/local/bin/kube-scheduler \ --v=2 \ --logtostderr=true \ --address=127.0.0.1 \ --leader-elect=true \ --kubeconfig=/etc/kubernetes/scheduler.kubeconfig  Restart=always RestartSec=10s  [Install] WantedBy=multi-user.target

启动

 systemctl daemon-reload && systemctl enable --now kube-scheduler

查看状态

systemctl status kube-scheduler

TLS Bootstrapping配置

Kubernetes 在 1.4 版本(我记着是)推出了 TLS bootstrapping 功能；这个功能主要解决了以下问题:

当集群开启了 TLS 认证后，每个节点的 kubelet 组件都要使用由 apiserver 使用的 CA 签发的有效证书才能与 apiserver 通讯；此时如果节点多起来，为每个节点单独签署证书将是一件非常繁琐的事情；

TLS bootstrapping 功能就是让 kubelet 先使用一个预定的低权限用户连接到 apiserver，然后向 apiserver 申请证书，kubelet 的证书由 apiserver 动态签署；在配合 RBAC 授权模型下的工作

创建bootstrap（Master01节点）

注意，如果不是高可用集群，192.168.32.233:8443改为master01的地址，8443改为apiserver的端口，默认是6443

[root@k8s-master01 ~]# cd /root/k8s-ha-install/bootstrap  kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.pem --embed-certs=true --server=https://192.168.32.233:8443 --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig kubectl config set-credentials tls-bootstrap-token-user --token=c8ad9c.2e4d610cf3e7426e --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig kubectl config set-context tls-bootstrap-token-user@kubernetes --cluster=kubernetes --user=tls-bootstrap-token-user --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig kubectl config use-context tls-bootstrap-token-user@kubernetes --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

注意：如果要修改bootstrap.secret.yaml的token-id和token-secret，需保证各字段的对应关系，还有上面命令中的 --token 字段的对应

[root@k8s-master01 bootstrap]# mkdir -p /root/.kube ; cp /etc/kubernetes/admin.kubeconfig /root/.kube/config [root@k8s-master01 bootstrap]# kubectl create -f bootstrap.secret.yaml

Node节点配置

复制证书至Node（这里各master 也充当了node）节点，1.19版本以后，建议各master节点，把 kubelet,kube-proxy,都装上，占用不了多少资源，可以通过污点等方式，不运行业务pod就好了

cd /etc/kubernetes/  for NODE in k8s-master02 k8s-master03 node01 node02; do ssh $NODE mkdir -p /etc/kubernetes/pki /etc/etcd/ssl /etc/etcd/ssl for FILE in etcd-ca.pem etcd.pem etcd-key.pem; do scp /etc/etcd/ssl/$FILE $NODE:/etc/etcd/ssl/ done for FILE in pki/ca.pem pki/ca-key.pem pki/front-proxy-ca.pem bootstrap-kubelet.kubeconfig; do scp /etc/kubernetes/$FILE $NODE:/etc/kubernetes/${FILE} done done

Kubelet配置

创建相关目录（所有节点）

mkdir -p /var/lib/kubelet /var/log/kubernetes /etc/systemd/system/kubelet.service.d /etc/kubernetes/manifests/

配置kubelet service（所有节点）

vim /usr/lib/systemd/system/kubelet.service  [Unit] Description=Kubernetes Kubelet Documentation=https://github.com/kubernetes/kubernetes After=docker.service Requires=docker.service  [Service] ExecStart=/usr/local/bin/kubelet  Restart=always StartLimitInterval=0 RestartSec=10  [Install] WantedBy=multi-user.target

配置kubelet service的配置文件（所有节点）

vim /etc/systemd/system/kubelet.service.d/10-kubelet.conf  [Service] Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig" Environment="KUBELET_SYSTEM_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin" Environment="KUBELET_CONFIG_ARGS=--config=/etc/kubernetes/kubelet-conf.yml --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.2" Environment="KUBELET_EXTRA_ARGS=--node-labels=node.kubernetes.io/node='' " ExecStart= ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_SYSTEM_ARGS $KUBELET_EXTRA_ARGS

kubelet的配置文件（所有节点）启动所有节点kubelet

注意：如果更改了k8s的service网段，需要更改kubelet-conf.yml 的clusterDNS:配置，改成k8s Service网段的第十个地址，比如10.96.0.10（k8s的service网段开始设置的是10.96.0.0/12）

vim /etc/kubernetes/kubelet-conf.yml  apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration address: 0.0.0.0 port: 10250 readOnlyPort: 10255 authentication: anonymous: enabled: false webhook: cacheTTL: 2m0s enabled: true x509: clientCAFile: /etc/kubernetes/pki/ca.pem authorization: mode: Webhook webhook: cacheAuthorizedTTL: 5m0s cacheUnauthorizedTTL: 30s cgroupDriver: systemd cgroupsPerQOS: true clusterDNS: - 10.96.0.10 clusterDomain: cluster.local containerLogMaxFiles: 5 containerLogMaxSize: 10Mi contentType: application/vnd.kubernetes.protobuf cpuCFSQuota: true cpuManagerPolicy: none cpuManagerReconcilePeriod: 10s enableControllerAttachDetach: true enableDebuggingHandlers: true enforceNodeAllocatable: - pods eventBurst: 10 eventRecordQPS: 5 evictionHard: imagefs.available: 15% memory.available: 100Mi nodefs.available: 10% nodefs.inodesFree: 5% evictionPressureTransitionPeriod: 5m0s failSwapOn: true fileCheckFrequency: 20s hairpinMode: promiscuous-bridge healthzBindAddress: 127.0.0.1 healthzPort: 10248 httpCheckFrequency: 20s imageGCHighThresholdPercent: 85 imageGCLowThresholdPercent: 80 imageMinimumGCAge: 2m0s iptablesDropBit: 15 iptablesMasqueradeBit: 14 kubeAPIBurst: 10 kubeAPIQPS: 5 makeIPTablesUtilChains: true maxOpenFiles: 1000000 maxPods: 110 nodeStatusUpdateFrequency: 10s oomScoreAdj: -999 podPidsLimit: -1 registryBurst: 10 registryPullQPS: 5 resolvConf: /etc/resolv.conf rotateCertificates: true runtimeRequestTimeout: 2m0s serializeImagePulls: true staticPodPath: /etc/kubernetes/manifests streamingConnectionIdleTimeout: 4h0m0s syncFrequency: 1m0s volumeStatsAggPeriod: 1m0s

启动kubelet（所有节点）

systemctl daemon-reload systemctl enable --now kubelet

查看集群状态（matser01上）

[root@k8s-master01 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master01 NotReady <none> 6m19s v1.20.14 k8s-master02 NotReady <none> 5m57s v1.20.14 k8s-master03 NotReady <none> 5m57s v1.20.14 node01 NotReady <none> 6m48s v1.20.14 node02 NotReady <none> 6m48s v1.20.14

kube-proxy配置

注意，如果不是高可用集群，192.168.32.233:8443改为master01的地址，8443改为apiserver的端口，默认是6443

Master01执行

[root@k8s-master01 ~]# cd /root/k8s-ha-install  kubectl -n kube-system create serviceaccount kube-proxy kubectl create clusterrolebinding system:kube-proxy --clusterrole system:node-proxier --serviceaccount kube-system:kube-proxy SECRET=$(kubectl -n kube-system get sa/kube-proxy \ --output=jsonpath='{.secrets[0].name}') JWT_TOKEN=$(kubectl -n kube-system get secret/$SECRET \ --output=jsonpath='{.data.token}' | base64 -d) PKI_DIR=/etc/kubernetes/pki K8S_DIR=/etc/kubernetes kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.pem --embed-certs=true --server=https://192.168.32.233:8443 --kubeconfig=${K8S_DIR}/kube-proxy.kubeconfig kubectl config set-credentials kubernetes --token=${JWT_TOKEN} --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig kubectl config set-context kubernetes --cluster=kubernetes --user=kubernetes --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig kubectl config use-context kubernetes --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

发送kube-proxy的systemd Service文件发送到其他节点（master01上）

如果更改了集群Pod的网段，需要更改kube-proxy/kube-proxy.conf的clusterCIDR: 172.16.0.0/12参数为pod的网段

[root@k8s-master01 ~]# vim /root/k8s-ha-install/kube-proxy/kube-proxy.conf clusterCIDR: 172.16.0.0/12 #把这个字段更改为，自己的pod 网段

分发配置文件（master01上）

[root@k8s-master01 ~]# cd /root/k8s-ha-install for NODE in k8s-master01 k8s-master02 k8s-master03; do scp ${K8S_DIR}/kube-proxy.kubeconfig $NODE:/etc/kubernetes/kube-proxy.kubeconfig scp kube-proxy/kube-proxy.conf $NODE:/etc/kubernetes/kube-proxy.conf scp kube-proxy/kube-proxy.service $NODE:/usr/lib/systemd/system/kube-proxy.service done  for NODE in node01 node02; do scp /etc/kubernetes/kube-proxy.kubeconfig $NODE:/etc/kubernetes/kube-proxy.kubeconfig scp kube-proxy/kube-proxy.conf $NODE:/etc/kubernetes/kube-proxy.conf scp kube-proxy/kube-proxy.service $NODE:/usr/lib/systemd/system/kube-proxy.service done

启动kube-proxy（所有节点）

 systemctl daemon-reload && systemctl enable --now kube-proxy

安装Calico

[root@k8s-master01 ~]# cd /root/k8s-ha-install/calico/  # 修改calico-etcd.yaml的以下位置  sed -i 's#etcd_endpoints: "http://<ETCD_IP>:<ETCD_PORT>"#etcd_endpoints: "https://192.168.32.129:2379,https://192.168.32.130:2379,https://192.168.32.131:2379"#g' calico-etcd.yaml    ETCD_CA=`cat /etc/kubernetes/pki/etcd/etcd-ca.pem | base64 | tr -d '\n'` ETCD_CERT=`cat /etc/kubernetes/pki/etcd/etcd.pem | base64 | tr -d '\n'` ETCD_KEY=`cat /etc/kubernetes/pki/etcd/etcd-key.pem | base64 | tr -d '\n'`   sed -i "s@# etcd-key: null@etcd-key: ${ETCD_KEY}@g; s@# etcd-cert: null@etcd-cert: ${ETCD_CERT}@g; s@# etcd-ca: null@etcd-ca: ${ETCD_CA}@g" calico-etcd.yaml   sed -i 's#etcd_ca: ""#etcd_ca: "/calico-secrets/etcd-ca"#g; s#etcd_cert: ""#etcd_cert: "/calico-secrets/etcd-cert"#g; s#etcd_key: "" #etcd_key: "/calico-secrets/etcd-key" #g' calico-etcd.yaml   # 更改此处为自己的pod网段 POD_SUBNET="172.16.0.0/12"   sed -i 's@# - name: CALICO_IPV4POOL_CIDR@- name: CALICO_IPV4POOL_CIDR@g; s@# value: "192.168.0.0/16"@ value: '"${POD_SUBNET}"'@g' calico-etcd.yaml

apply

[root@k8s-master01 calico]# kubectl apply -f calico-etcd.yaml

查看容器状态

[root@k8s-master01 calico]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-5f6d4b864b-6jp52 1/1 Running 0 3m27s calico-node-2glmr 1/1 Running 0 3m26s calico-node-bvfkj 1/1 Running 0 3m26s calico-node-ffbwq 1/1 Running 0 3m26s calico-node-tpk2b 1/1 Running 0 3m26s calico-node-vv9r7 1/1 Running 0 3m26s [root@k8s-master01 calico]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master01 Ready <none> 43m v1.20.14 k8s-master02 Ready <none> 43m v1.20.14 k8s-master03 Ready <none> 43m v1.20.14 node01 Ready <none> 43m v1.20.14 node02 Ready <none> 43m v1.20.14

安装CoreDNS

如果更改了k8s service的网段需要将coredns的serviceIP改成k8s service网段的第十个IP

cd /root/k8s-ha-install/  sed -i "s#10.96.0.10#x.x.x.10#g" CoreDNS/coredns.yaml

安装coredns

[root@k8s-master01 k8s-ha-install]# kubectl create -f CoreDNS/coredns.yaml

安装Metrics Server

在新版的Kubernetes中系统资源的采集均使用Metrics-server，可以通过Metrics采集节点和Pod的内存、磁盘、CPU和网络的使用率

[root@k8s-master01 ~]# cd /root/k8s-ha-install/metrics-server-0.4.x/ [root@k8s-master01 metrics-server-0.4.x]# kubectl create -f .

安装dashboard

Dashboard用于展示集群中的各类资源，同时也可以通过Dashboard实时查看Pod的日志和在容器中执行一些命令等。

安装指定版本dashboard

[root@k8s-master01 ~]# cd /root/k8s-ha-install/dashboard/  [root@k8s-master01 dashboard]# kubectl create -f .

登录dashboard

# 更改dashboard的svc为NodePort [root@k8s-master01 ~]# kubectl edit svc kubernetes-dashboard -n kubernetes-dashboard 将ClusterIP更改为NodePort  # 查看端口号 [root@k8s-master01 dashboard]# kubectl get svc kubernetes-dashboard -n kubernetes-dashboard NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes-dashboard NodePort 10.107.11.77 <none> 443:31214/TCP 2m49s   # 根据自己的实例端口号，通过任意安装了kube-proxy的宿主机或者VIP的IP+端口即可访问到dashboard

页面访问：https://192.168.32.233:31214

查看token值

[root@k8s-master01 ~]# kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')

集群验证

安装busybox（master01上）

cat<<EOF | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: busybox namespace: default spec: containers: - name: busybox image: busybox:1.28 command: - sleep - "3600" imagePullPolicy: IfNotPresent restartPolicy: Always EOF

验证步骤（matser01上）

1. Pod必须能解析Service 2. Pod必须能解析跨namespace的Service 3. 每个节点都必须要能访问Kubernetes的kubernetes svc 443和kube-dns的service 53 4. Pod和Pod之前要能通 a) 同namespace能通信 b) 跨namespace能通信 c) 跨机器能通信

步骤演示（matser01上）

# 首先查看po是否安装成功 [root@k8s-master01 ~]# kubectl get po NAME READY STATUS RESTARTS AGE busybox 1/1 Running 0 3m11s  # 查看svc是否正常 [root@k8s-master01 ~]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 163m  # 查看Pod是否能能解析Service [root@k8s-master01 ~]# kubectl exec busybox -n default -- nslookup kubernetes Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local  Name: kubernetes Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local  # 查看Pod是否能解析跨namespace的Service [root@k8s-master01 ~]# kubectl exec busybox -n default -- nslookup kube-dns.kube-system Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local  Name: kube-dns.kube-system Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local  # 跟我以上结果一致就成功了

使用telnet命令验证

# 所有节点安装telnet命令,有的话忽略 yum install -y telnet  # 所有机器 10.96.0.1 443 kubernetes svc 443 # 所有机器 10.96.0.10 53 kube-dns的service 53 # 不会自动断开就是成功了 telnet 10.96.0.1 443 telnet 10.96.0.10 53  Trying 10.96.0.1... Connected to 10.96.0.1. Escape character is '^]'.

使用curl命令验证（所有机器）

[root@k8s-master01 ~]# curl 10.96.0.10:53 curl: (52) Empty reply from server

容器验证（master01上）

[root@k8s-master01 ~]# kubectl get po -n kube-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-5f6d4b864b-pq2qw 1/1 Running 0 62m calico-node-75blv 1/1 Running 0 62m calico-node-hw27b 1/1 Running 0 62m calico-node-k2wdf 1/1 Running 0 62m calico-node-l58lz 1/1 Running 0 62m calico-node-v2qlq 1/1 Running 0 62m coredns-867d46bfc6-8vzrk 1/1 Running 0 72m metrics-server-595f65d8d5-kgn8c 1/1 Running 0 60m  [root@k8s-master01 ~]# kubectl get po -n kube-system -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES calico-kube-controllers-5f6d4b864b-pq2qw 1/1 Running 0 63m 192.168.1.100 k8s-master01 <none> <none> calico-node-75blv 1/1 Running 0 63m 192.168.1.103 k8s-node01 <none> <none> calico-node-hw27b 1/1 Running 0 63m 192.168.1.101 k8s-master02 <none> <none> calico-node-k2wdf 1/1 Running 0 63m 192.168.1.100 k8s-master01 <none> <none> calico-node-l58lz 1/1 Running 0 63m 192.168.1.102 k8s-master03 <none> <none> calico-node-v2qlq 1/1 Running 0 63m 192.168.1.104 k8s-node02 <none> <none> coredns-867d46bfc6-8vzrk 1/1 Running 0 73m 172.161.125.2 k8s-node01 <none> <none> metrics-server-595f65d8d5-kgn8c 1/1 Running 0 62m 172.161.125.1 k8s-node01 <none> <none>  # 能进去就ok [root@k8s-master01 ~]# kubectl exec -it calico-node-v2qlq -n kube-system -- sh sh-4.4#  # 进入node01，然后能ping通node02就行 [root@k8s-master01 ~]# kubectl exec -it calico-node-v2qlq -n kube-system -- bash [root@k8s-node02 /]# ping 192.168.1.104 PING 192.168.1.104 (192.168.1.104) 56(84) bytes of data. 64 bytes from 192.168.1.104: icmp_seq=1 ttl=64 time=0.123 ms 64 bytes from 192.168.1.104: icmp_seq=2 ttl=64 time=0.090 ms ^C --- 192.168.1.104 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 46ms rtt min/avg/max/mdev = 0.090/0.106/0.123/0.019 ms

常用配置

# 所有节点都改 vim /etc/docker/daemon.json  { "registry-mirrors": ["https://6h6ezoe5.mirror.aliyuncs.com"], "exec-opts": ["native.cgroupdriver=systemd"], "max-concurrent-downloads": 10, "max-concurrent-uploads": 5, "log-opts": { "max-size": "300m", "max-file": "2" }, "live-restore": true }  # 所有节点改完重启docker systemctl daemon-reload && systemctl restart docker   max-concurrent-downloads # 下载并发数,最大开启线程下载镜像 max-concurrent-uploads # 上传并发数，同上 max-size # 日志文件最大到多少切割 max-file # 日志文件保留个数 live-restore # 开启这个参数，重启docker，不会导致 容器重启

vim /usr/lib/systemd/system/kube-controller-manager.service # 找个位置加上,在三个master节点 --experimental-cluster-signing-duration=876000h0m0s \ # 改完重启 systemctl daemon-reload && systemctl restart kube-controller-manager ##TLS bootstrapping 时的证书实际是由 kube-controller-manager 组件来签署的，也就是说证书有效期是 kube-controller-manager 组件控制的 默认为 8760h0m0s，将其改为 87600h0m0s 即 10 年后再进行 TLS bootstrapping 签署证书即可 bootstrapping 会在证书快要过期的时候自动续签

# 所有节点，更换成以下的配置文件 [root@k8s-node02 ~]# cat /etc/systemd/system/kubelet.service.d/10-kubelet.conf  [Service] Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig" Environment="KUBELET_SYSTEM_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin" Environment="KUBELET_CONFIG_ARGS=--config=/etc/kubernetes/kubelet-conf.yml --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.2" Environment="KUBELET_EXTRA_ARGS=--node-labels=node.kubernetes.io/node='' --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --image-pull-progress-deadline=30m " ExecStart= ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_SYSTEM_ARGS $KUBELET_EXTRA_ARGS    #增加了 --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 安全加固 #增加了 --image-pull-progress-deadline=30m #镜像拉取进度最大时间，如果在这段时间拉取镜像没有任何进展，将取消拉取

以上内容由“WiFi之家网”整理收藏！。

原创文章，作者：192.168.1.1，如若转载，请注明出处：https://www.224m.com/232163.html