行业知识
第3关 二进制形式安装K8s高可用生产级集群
2026-11-07 14:21:52
来源：互联网转载
------> ��程视频同步分享在今日头条和B站
大家好，我是博哥爱运维，下面是这次安装k8s集群相关系统及组件的详细版本号
- Ubuntu 22.04.3 LTS
- k8s: v1.27.5
- containerd: 1.6.23
- etcd: v3.5.9
- coredns: 1.11.1
- calico: v3.24.6
下面是此次虚拟机集群安装前的IP等信息规划(完全模拟一个中小型企业K8S集群)
IP hostname role resource 10.0.1.201 node-1 master/work node 2c/4g(ingress-nginx) 10.0.1.202 node-2 master/work node 2c/4g(harbor) 10.0.1.203 node-3 work node 2c/4g 10.0.1.204 node-4 work node 2c/4g 显然目前为止，前面几关给我们的装备还不太够，我们继续在这一关获取充足的装备弹药，为最终战胜K8s而奋斗吧！
这里采用开源项目https://github.com/easzlab/kubeasz，以二进制安装的方式，符合企业级稳定运行的要求，非常优秀的一个开源项目，亲眼见证了star数从0到现在最新的9.6k的增长过程，版本也在不停地迭代更新，基本是跟着K8S的更新节奏来。
博哥在实际工作中，在全球各区域（国内、北美、欧洲、东南亚），各个主流云厂商（阿里云、华为云、腾讯云、百度云、火山云、移动云、AWS、Google cloud等），均自建过生产K8S集群提供业务服务能力，用的就是这个优秀的K8S安装开源项目，稳定运行时间长的集群有4年左右，期间没出现过大问题很稳定。
部署架构图
安装步骤清单：
- deploy机器做好对所有k8s node节点的免密登陆操作
- deploy机器安装好python2版本以及pip，然后安装ansible
- 对k8s集群配置做一些定制化配置并开始部署
# 需要注意的在线安装因为会从github及dockerhub上下载文件及镜像，有时候访问这些国外网络会非常慢，这里我也会大家准备好了完整离线安装包，下载地址如下，和上面的安装脚本放在同一目录下，再执行上面的安装命令即可# 此离线安装包里面的k8s版本为v1.27.5https://cloud.189.cn/web/share?code=Yj6nEv6jQ3Mv（访问码：3pms）
自建k8s集群部署
挂载数据盘
注意： 如无需独立数据盘可忽略此步骤
# 创建下面4个目录/var/lib/container/{kubelet,docker,nfs_dir}/nfs_dir# 不分区直接格式化数据盘，假设数据盘是/dev/vdbmkfs.ext4 /dev/vdb# 然后编辑 /etc/fstab，添加如下内容：/dev/vdb /var/lib/container/ ext4 defaults 0 0/var/lib/container/kubelet /var/lib/kubelet none defaults,bind 0 0/var/lib/container/docker /var/lib/docker none defaults,bind 0 0/var/lib/container/nfs_dir /nfs_dir none defaults,bind 0 0# 刷新生效挂载mount -a自己编写的k8s安装脚本说明
部署脚本调用核心项目github: https://github.com/easzlab/kubeasz ， 此脚本是这个项目的上一层简化二进制部署k8s实施的封装
此脚本安装过的操作系统 CentOS 7, Ubuntu 16.04/18.04/20.04/22.04
注意: k8s 版本 >= 1.24 时，CRI仅支持 containerd
# 安装命令示例（假设我这里root的密码是rootPassword，如已做免密这里的密码可以任意填写；10.0.1为内网网段；后面的依次是主机位；CRI容器运行时；CNI网络插件；我们自己的域名是boge.com；要设定k8s集群名称为test）：# 单台节点部署bash k8s_install_new.sh rootPassword 10.0.1 201 containerd calico boge.com test# 多台节点部署bash k8s_install_new.sh rootPassword 10.0.1 201\ 202\ 203\ 204 containerd calico boge.com test# 注意：如果是在海外部署，而集群名称又不带aws的话，可以把安装脚本内此部分代码注释掉，避免pip安装过慢if ! `echo $clustername |grep -iwE aws &>/dev/null`; thenmkdir ~/.pipcat > ~/.pip/pip.conf /dev/null;then software_packet="$(ls -1v ./kubeasz*.tar.gz )";else software_packet="";fipwd="/etc/kubeasz"# deploy机器升级软件库if cat /etc/redhat-release &>/dev/null;then yum update -yelse apt-get update && apt-get upgrade -y && apt-get dist-upgrade -y [ $? -ne 0 ] && apt-get -yf installfi# deploy机器检测python环境python2 -V &>/dev/nullif [ $? -ne 0 ];then if cat /etc/redhat-release &>/dev/null;then yum install gcc openssl-devel bzip2-devel wget https://www.python.org/ftp/python/2.7.16/Python-2.7.16.tgz tar xzf Python-2.7.16.tgz cd Python-2.7.16 ./configure --enable-optimizations make altinstall ln -s /usr/bin/python2.7 /usr/bin/python cd - else apt-get install -y python2.7 && ln -s /usr/bin/python2.7 /usr/bin/python fifipython3 -V &>/dev/nullif [ $? -ne 0 ];then if cat /etc/redhat-release &>/dev/null;then yum install python3 -y else apt-get install -y python3 fifi# deploy机器设置pip安装加速源if `echo $clustername |grep -iwE cn &>/dev/null`; thenmkdir ~/.pipcat > ~/.pip/pip.conf wget https://bootstrap.pypa.io/pip/2.7/get-pip.py && python get-pip.py }else if grep -Ew '20.04|22.04' /etc/issue &/dev/null;then apt-get install sshpass -y;else apt-get install python-pip sshpass -y;fi [ -f ./get-pip.py ] && python ./get-pip.py || { wget https://bootstrap.pypa.io/pip/2.7/get-pip.py && python get-pip.py }fipython -m pip install --upgrade "pip /dev/null;then ssh -o StrictHostKeyChecking=no ${USER}@${netnum}.${host} "yum update -y" else ssh -o StrictHostKeyChecking=no ${USER}@${netnum}.${host} "apt-get update && apt-get upgrade -y && apt-get dist-upgrade -y" [ $? -ne 0 ] && ssh -o StrictHostKeyChecking=no ${USER}@${netnum}.${host} "apt-get -yf install" fidone# deploy机器下载k8s二进制安装脚本(注：这里下载可能会因网络原因失败，可以多尝试运行该脚本几次)if [[ ${software_packet} == '' ]];then if [[ ! -f ./ezdown ]];then curl -C- -fLO --retry 3 https://github.com/easzlab/kubeasz/releases/download/${release}/ezdown fi # 使用工具脚本下载 sed -ri "s+^(K8S_BIN_VER=).*$+${k8s_ver}+g" ezdown chmod +x ./ezdown # ubuntu_22 to download package of Ubuntu 22.04 ./ezdown -D && ./ezdown -P ubuntu_22 && ./ezdown -Xelse tar xvf ${software_packet} -C /etc/ sed -ri "s+^(K8S_BIN_VER=).*$+${k8s_ver}+g" ${pwd}/ezdown chmod +x ${pwd}/{ezctl,ezdown} chmod +x ./ezdown ./ezdown -D # 离线安装 docker，检查本地文件，正常会提示所有文件已经下载完成，并上传到本地私有镜像仓库 ./ezdown -S # 启动 kubeasz 容器fi# 初始化一个名为$clustername的k8s集群配置CLUSTER_NAME="$clustername"${pwd}/ezctl new ${CLUSTER_NAME}if [[ $? -ne 0 ]];then echo "cluster name [${CLUSTER_NAME}] was exist in ${pwd}/clusters/${CLUSTER_NAME}." exit 1fiif [[ ${software_packet} != '' ]];then # 设置参数，启用离线安装 # 离线安装文档：https://github.com/easzlab/kubeasz/blob/3.6.2/docs/setup/offline_install.md sed -i 's/^INSTALL_SOURCE.*$/INSTALL_SOURCE: "offline"/g' ${pwd}/clusters/${CLUSTER_NAME}/config.ymlfi# to check ansible serviceansible all -m ping#---------------------------------------------------------------------------------------------------#修改二进制安装脚本配置 config.ymlsed -ri "s+^(CLUSTER_NAME:).*$+ \"${CLUSTER_NAME}\"+g" ${pwd}/clusters/${CLUSTER_NAME}/config.yml## k8s上日志及容器数据存独立磁盘步骤（参考阿里云的）mkdir -p /var/lib/container/{kubelet,docker,nfs_dir} /var/lib/{kubelet,docker} /nfs_dir## 不用fdisk分区，直接格式化数据盘 mkfs.ext4 /dev/vdb，按下面添加到fstab后，再mount -a刷新挂载(blkid /dev/sdx)## cat /etc/fstab # UUID=105fa8ff-bacd-491f-a6d0-f99865afc3d6 / ext4 defaults 1 1# /dev/vdb /var/lib/container/ ext4 defaults 0 0# /var/lib/container/kubelet /var/lib/kubelet none defaults,bind 0 0# /var/lib/container/docker /var/lib/docker none defaults,bind 0 0# /var/lib/container/nfs_dir /nfs_dir none defaults,bind 0 0## tree -L 1 /var/lib/container# /var/lib/container# ├── docker# ├── kubelet# └── lost+found# docker data dirDOCKER_STORAGE_DIR="/var/lib/container/docker"sed -ri "s+^(STORAGE_DIR:).*$+STORAGE_DIR: \"${DOCKER_STORAGE_DIR}\"+g" ${pwd}/clusters/${CLUSTER_NAME}/config.yml# containerd data dirCONTAINERD_STORAGE_DIR="/var/lib/container/containerd"sed -ri "s+^(STORAGE_DIR:).*$+STORAGE_DIR: \"${CONTAINERD_STORAGE_DIR}\"+g" ${pwd}/clusters/${CLUSTER_NAME}/config.yml# kubelet logs dirKUBELET_ROOT_DIR="/var/lib/container/kubelet"sed -ri "s+^(KUBELET_ROOT_DIR:).*$+KUBELET_ROOT_DIR: \"${KUBELET_ROOT_DIR}\"+g" ${pwd}/clusters/${CLUSTER_NAME}/config.ymlif [[ $clustername != 'aws' ]]; then # docker aliyun repo REG_MIRRORS="https://pqbap4ya.mirror.aliyuncs.com" sed -ri "s+^REG_MIRRORS:.*$+REG_MIRRORS: \'[\"${REG_MIRRORS}\"]\'+g" ${pwd}/clusters/${CLUSTER_NAME}/config.ymlfi# [docker]信任的HTTP仓库sed -ri "s+127.0.0.1/8+${netnum}.0/24+g" ${pwd}/clusters/${CLUSTER_NAME}/config.yml# disable dashboard auto installsed -ri "s+^(dashboard_install:).*$+ \"no\"+g" ${pwd}/clusters/${CLUSTER_NAME}/config.yml# 融合配置准备(按示例部署命令这里会生成testk8s.boge.com这个域名，部署脚本会基于这个域名签证书，优势是后面访问kube-apiserver，可以基于此域名解析任意IP来访问，灵活性更高)CLUSEER_WEBSITE="${CLUSTER_NAME}k8s.${domainName}"lb_num=$(grep -wn '^MASTER_CERT_HOSTS:' ${pwd}/clusters/${CLUSTER_NAME}/config.yml |awk -F: '{print }')lb_num1=$(expr ${lb_num} + 1)lb_num2=$(expr ${lb_num} + 2)sed -ri "${lb_num1}s+.*$+ - "${CLUSEER_WEBSITE}"+g" ${pwd}/clusters/${CLUSTER_NAME}/config.ymlsed -ri "${lb_num2}s+(.*)$+#+g" ${pwd}/clusters/${CLUSTER_NAME}/config.yml# node节点最大pod 数MAX_PODS="120"sed -ri "s+^(MAX_PODS:).*$+ ${MAX_PODS}+g" ${pwd}/clusters/${CLUSTER_NAME}/config.yml# calico 自建机房都在二层网络可以设置 CALICO_IPV4POOL_IPIP=“off”,以提高网络性能; 公有云上VPC在三层网络，需设置CALICO_IPV4POOL_IPIP: "Always"开启ipip隧道#sed -ri "s+^(CALICO_IPV4POOL_IPIP:).*$+ \"off\"+g" ${pwd}/clusters/${CLUSTER_NAME}/config.yml# 修改二进制安装脚本配置 hosts# clean old ipsed -ri '/192.168.1.1/d' ${pwd}/clusters/${CLUSTER_NAME}/hostssed -ri '/192.168.1.2/d' ${pwd}/clusters/${CLUSTER_NAME}/hostssed -ri '/192.168.1.3/d' ${pwd}/clusters/${CLUSTER_NAME}/hostssed -ri '/192.168.1.4/d' ${pwd}/clusters/${CLUSTER_NAME}/hostssed -ri '/192.168.1.5/d' ${pwd}/clusters/${CLUSTER_NAME}/hosts# 输入准备创建ETCD集群的主机位echo "enter etcd hosts here (example: 203 202 201) ↓"read -p "" ipnumsfor ipnum in `echo ${ipnums}`do echo $netnum.$ipnum sed -i "/\[etcd/a $netnum.$ipnum" ${pwd}/clusters/${CLUSTER_NAME}/hostsdone# 输入准备创建KUBE-MASTER集群的主机位echo "enter kube-master hosts here (example: 202 201) ↓"read -p "" ipnumsfor ipnum in `echo ${ipnums}`do echo $netnum.$ipnum sed -i "/\[kube_master/a $netnum.$ipnum" ${pwd}/clusters/${CLUSTER_NAME}/hostsdone# 输入准备创建KUBE-NODE集群的主机位echo "enter kube-node hosts here (example: 204 203) ↓"read -p "" ipnumsfor ipnum in `echo ${ipnums}`do echo $netnum.$ipnum sed -i "/\[kube_node/a $netnum.$ipnum" ${pwd}/clusters/${CLUSTER_NAME}/hostsdone# 配置容器运行时CNIcase ${cni} in flannel) sed -ri "s+^CLUSTER_NETWORK=.*$+CLUSTER_NETWORK=\"${cni}\"+g" ${pwd}/clusters/${CLUSTER_NAME}/hosts ;; calico) sed -ri "s+^CLUSTER_NETWORK=.*$+CLUSTER_NETWORK=\"${cni}\"+g" ${pwd}/clusters/${CLUSTER_NAME}/hosts ;; cilium) sed -ri "s+^CLUSTER_NETWORK=.*$+CLUSTER_NETWORK=\"${cni}\"+g" ${pwd}/clusters/${CLUSTER_NAME}/hosts ;; *) echo "cni need be flannel or calico or cilium." exit 11esac# 配置K8S的ETCD数据备份的定时任务# https://github.com/easzlab/kubeasz/blob/master/docs/op/cluster_restore.mdif cat /etc/redhat-release &>/dev/null;then if ! grep -w '94.backup.yml' /var/spool/cron/root &>/dev/null;then echo "00 00 * * * /usr/local/bin/ansible-playbook -i /etc/kubeasz/clusters/${CLUSTER_NAME}/hosts -e @/etc/kubeasz/clusters/${CLUSTER_NAME}/config.yml /etc/kubeasz/playbooks/94.backup.yml &> /dev/null; find /etc/kubeasz/clusters/${CLUSTER_NAME}/backup/ -type f -name '*.db' -mtime +3|xargs rm -f" >> /var/spool/cron/root;else echo exists ;fi chown root.crontab /var/spool/cron/root chmod 600 /var/spool/cron/root rm -f /var/run/cron.reboot service crond restartelse if ! grep -w '94.backup.yml' /var/spool/cron/crontabs/root &>/dev/null;then echo "00 00 * * * /usr/local/bin/ansible-playbook -i /etc/kubeasz/clusters/${CLUSTER_NAME}/hosts -e @/etc/kubeasz/clusters/${CLUSTER_NAME}/config.yml /etc/kubeasz/playbooks/94.backup.yml &> /dev/null; find /etc/kubeasz/clusters/${CLUSTER_NAME}/backup/ -type f -name '*.db' -mtime +3|xargs rm -f" >> /var/spool/cron/crontabs/root;else echo exists ;fi chown root.crontab /var/spool/cron/crontabs/root chmod 600 /var/spool/cron/crontabs/root rm -f /var/run/crond.reboot service cron restartfi#---------------------------------------------------------------------------------------------------# 准备开始安装了rm -rf ${pwd}/{dockerfiles,docs,.gitignore,pics,dockerfiles} &&\find ${pwd}/ -name '*.md'|xargs rm -fread -p "Enter to continue deploy k8s to all nodes >>>" YesNobbb# now start deploy k8s cluster cd ${pwd}/# to prepare CA/certs & kubeconfig & other system settings ${pwd}/ezctl setup ${CLUSTER_NAME} 01sleep 1# to setup the etcd cluster${pwd}/ezctl setup ${CLUSTER_NAME} 02sleep 1# to setup the container runtime(docker or containerd)case ${cri} in containerd) sed -ri "s+^CONTAINER_RUNTIME=.*$+CONTAINER_RUNTIME=\"${cri}\"+g" ${pwd}/clusters/${CLUSTER_NAME}/hosts ${pwd}/ezctl setup ${CLUSTER_NAME} 03 ;; docker) sed -ri "s+^CONTAINER_RUNTIME=.*$+CONTAINER_RUNTIME=\"${cri}\"+g" ${pwd}/clusters/${CLUSTER_NAME}/hosts ${pwd}/ezctl setup ${CLUSTER_NAME} 03 ;; *) echo "cri need be containerd or docker." exit 11esacsleep 1# to setup the master nodes${pwd}/ezctl setup ${CLUSTER_NAME} 04sleep 1# to setup the worker nodes${pwd}/ezctl setup ${CLUSTER_NAME} 05sleep 1# to setup the network plugin(flannel、calico...)${pwd}/ezctl setup ${CLUSTER_NAME} 06sleep 1# to setup other useful plugins(metrics-server、coredns...)${pwd}/ezctl setup ${CLUSTER_NAME} 07sleep 1# [可选]对集群所有节点进行操作系统层面的安全加固 https://github.com/dev-sec/ansible-os-hardening#ansible-playbook roles/os-harden/os-harden.yml#sleep 1#cd `dirname ${software_packet:-/tmp}`k8s_bin_path='/opt/kube/bin'echo "------------------------- k8s version list ---------------------------"${k8s_bin_path}/kubectl versionechoecho "------------------------- All Healthy status check -------------------"${k8s_bin_path}/kubectl get componentstatusechoecho "------------------------- k8s cluster info list ----------------------"${k8s_bin_path}/kubectl cluster-infoechoecho "------------------------- k8s all nodes list -------------------------"${k8s_bin_path}/kubectl get node -o wideechoecho "------------------------- k8s all-namespaces's pods list ------------"${k8s_bin_path}/kubectl get pod --all-namespacesechoecho "------------------------- k8s all-namespaces's service network ------"${k8s_bin_path}/kubectl get svc --all-namespacesechoecho "------------------------- k8s welcome for you -----------------------"echo# you can use k alias kubectl to siampleecho "alias k=kubectl && complete -F __start_kubectl k" >> ~/.bashrc# get dashboard url${k8s_bin_path}/kubectl cluster-info|grep dashboard|awk '{print $NF}'|tee -a /root/k8s_results# get login token${k8s_bin_path}/kubectl -n kube-system describe secret $(${k8s_bin_path}/kubectl -n kube-system get secret | grep admin-user | awk '{print }')|grep 'token:'|awk '{print $NF}'|tee -a /root/k8s_resultsechoecho "you can look again dashboard and token info at >>> /root/k8s_results >>>>>>>>>>>>>> You need to excute command [ reboot ] to restart all nodes
上一篇：云服务器怎么发布网页?
下一篇：服务器主板型号怎么看?
