主要内容
简介
k8s全称kubernetes，是由google开发的容器编排引擎，和docker原生的swarm类似，但功能更强大。
安装相关组件
使用的系统是Ubuntu 16.04
安装和部署kubernetes需要以下组件：
- kubeadm
- kubectl
- kubelet
- Docker
使用kubeadm安装部署K8S十分简单，但需要注意的东西也比较多。由于K8S是google开发的，相关的Docker镜像也存放在Google自家的网站上面，墙内很可能出现无法拉取镜像的情况。
安装Docker
Docker的安装十分简单，只要跟着官方的教程即可，这里不在叙述。
安装命令行工具
这些工具包括kubeadm、kubectl和kubelet，如果使用K8S官方的教程安装，则需要翻墙才能装上，因此这里使用阿里云提供的仓库进行安装
- 添加阿里云仓库
12345678# 添加gpgcurl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -# 添加软件源cat <<EOF >/etc/apt/sources.list.d/kubernetes.listdeb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial mainEOF
- 安装工具
1234567apt updateapt install kubeadm kubelet kubectl# 更新时忽略这些软件包apt-mark hold kubeadm kubelet kubectl
准备工作基本完毕。
部署K8S
我们选择使用kubeadm命令创建K8S集群，这种方式是最简单的。
初始化master节点
关闭swap分区
由于K8S不支持使用交换分区，因此要先把交换分区关闭
|
1 2 |
swapoff -a |
还要把/etc/fstab文件中的相关设备注释，避免系统重启重新挂载交换分区。
如果忘记关闭swap，初始化过程会出现下面的错误
|
1 2 3 |
[ERROR Swap]: running with swap on is not supported. Please disable swap [preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=... |
kubeadm init命令
前面我们说过，K8S使用的相关镜像会从Google的网站上拉取，因此墙内会拉取失败。例如出现下面的错误
|
1 2 3 4 5 6 7 8 9 10 |
[preflight] Some fatal errors occurred: [ERROR ImagePull]: failed to pull image [k8s.gcr.io/kube-apiserver-amd64:v1.11.3]: exit status 1 [ERROR ImagePull]: failed to pull image [k8s.gcr.io/kube-controller-manager-amd64:v1.11.3]: exit status 1 [ERROR ImagePull]: failed to pull image [k8s.gcr.io/kube-scheduler-amd64:v1.11.3]: exit status 1 [ERROR ImagePull]: failed to pull image [k8s.gcr.io/kube-proxy-amd64:v1.11.3]: exit status 1 [ERROR ImagePull]: failed to pull image [k8s.gcr.io/pause:3.1]: exit status 1 [ERROR ImagePull]: failed to pull image [k8s.gcr.io/etcd-amd64:3.2.18]: exit status 1 [ERROR ImagePull]: failed to pull image [k8s.gcr.io/coredns:1.1.3]: exit status 1 [preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=... |
为了避免这些问题，我们的做法是使用docker hub上面的Google Mirror，先把需要的镜像pull下面，然后再手动打tag。
查看需要的镜像
拉取之前我们需要知道哪些镜像是K8S依赖的，可以通过下面的命令查看
|
1 2 3 4 5 6 7 8 9 |
root@server:~# kubeadm config images list --kubernetes-version v1.12.0 k8s.gcr.io/kube-apiserver:v1.12.0 k8s.gcr.io/kube-controller-manager:v1.12.0 k8s.gcr.io/kube-scheduler:v1.12.0 k8s.gcr.io/kube-proxy:v1.12.0 k8s.gcr.io/pause:3.1 k8s.gcr.io/etcd:3.2.24 k8s.gcr.io/coredns:1.2.2 |
一共是7个镜像，因此我们先从docker hub获取它们
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
#!/bin/bash version=1.12.0 images=$(kubeadm config images list --kubernetes-version v${version}) for image in ${images} do # 获取镜像名称 commonName=$(echo ${image} | awk -F'[/]' '{print $2}') # 墙内镜像地址 innerImage=anjia0532/google-containers.${commonName} #墙外镜像地址 outerImage=k8s.gcr.io/${commonName} docker pull ${innerImage} #重命名镜像名称 docker tag ${innerImage} ${outerImage} docker rmi ${innerImage} done |
上面的脚本有三个作用：
- 从docker hub拉取镜像
- 为镜像重新打tag
- 删除旧的镜像
注意：上面的部分镜像，其他节点同样需要使用，因此这个脚本需要在所有节点下面执行
创建master
所有依赖的镜像都拉取到本地后，就可以执行kubeadm init命令初始化master节点。
|
1 2 3 4 |
kubeadm init --apiserver-advertise-address 192.168.11.161 \ --pod-network-cidr 10.244.0.0/16 \ --kubernetes-version v1.12.0 |
由于我们以后会使用flannel的网络，因此--pod-network-cidr参数必须是10.244.0.0/16。
注意：如果--apiserver-advertise-address参数设置的地址不是本地地址（例如不小心打错了），那么master会创建失败
命令执行过程中会出现下面的执行记录
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 |
[init] using Kubernetes version: v1.12.0 [preflight] running pre-flight checks [preflight/images] Pulling images required for setting up a Kubernetes cluster [preflight/images] This might take a minute or two, depending on the speed of your internet connection [preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull' [kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [preflight] Activating the kubelet service [certificates] Generated ca certificate and key. [certificates] Generated apiserver certificate and key. [certificates] apiserver serving cert is signed for DNS names [server kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.11.161] [certificates] Generated apiserver-kubelet-client certificate and key. [certificates] Generated front-proxy-ca certificate and key. [certificates] Generated front-proxy-client certificate and key. [certificates] Generated etcd/ca certificate and key. [certificates] Generated apiserver-etcd-client certificate and key. [certificates] Generated etcd/server certificate and key. [certificates] etcd/server serving cert is signed for DNS names [server localhost] and IPs [127.0.0.1 ::1] [certificates] Generated etcd/peer certificate and key. [certificates] etcd/peer serving cert is signed for DNS names [server localhost] and IPs [192.168.11.161 127.0.0.1 ::1] [certificates] Generated etcd/healthcheck-client certificate and key. [certificates] valid certificates and keys now exist in "/etc/kubernetes/pki" [certificates] Generated sa key and public key. [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf" [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf" [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf" [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf" [controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml" [controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml" [controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml" [etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml" [init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests" [init] this might take a minute or longer if the control plane images have to be pulled [apiclient] All control plane components are healthy after 44.504438 seconds [uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config-1.12" in namespace kube-system with the configuration for the kubelets in the cluster [markmaster] Marking the node server as master by adding the label "node-role.kubernetes.io/master=''" [markmaster] Marking the node server as master by adding the taints [node-role.kubernetes.io/master:NoSchedule] [patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "server" as an annotation [bootstraptoken] using token: lh3ffu.9yr2wml1ahetaz7l [bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy Your Kubernetes master has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of machines by running the following on each node as root: kubeadm join 192.168.11.161:6443 --token lh3ffu.9yr2wml1ahetaz7l --discovery-token-ca-cert-hash sha256:37cdfea7ff4e2da10571a02610d49d4de3c37770a4e720c6a84f124302a18551 |
看到Your Kubernetes master has initialized successfully!，证明master创建成功。
配置master
master创建成功后，日志中已经提示了我们如何配置master和加入其他节点
根据提示，我们需要先创建一个普通用户
|
1 2 3 4 5 |
mkdir -p /home/k8s #创建用户的home目录 useradd k8s --home-dir /home/k8s #创建用户 passwd k8s #更新用户密码 chown k8s:k8s /home/k8s # 更改目录的所有权 |
然后根据提示配置master
|
1 2 3 4 |
mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config |
查看master状态
我们先来看一下master的状态
|
1 2 3 4 |
k8s@server:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION server NotReady master 10m v1.12.0 |
发现它的状态是NotReady，为什么呢？可以通过查看它依赖的容器目前的状态
|
1 2 3 4 5 6 7 8 9 10 |
k8s@server:~$ kubectl get pod --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-576cbf47c7-4ldh6 0/1 Pending 0 12m kube-system coredns-576cbf47c7-c4p6n 0/1 Pending 0 12m kube-system etcd-server 1/1 Running 0 12m kube-system kube-apiserver-server 1/1 Running 0 12m kube-system kube-controller-manager-server 1/1 Running 0 12m kube-system kube-proxy-tv6jp 1/1 Running 0 12m kube-system kube-scheduler-server 1/1 Running 0 12m |
有两个容器的状态是Pending，这就是造成master状态为NotReady的原因。那么为什么容器是pending呢？
通过查看syslog，发现一大堆类似如下错误的日志
|
1 2 3 |
Oct 5 03:49:05 server kubelet[70917]: W1005 03:49:05.970028 70917 cni.go:188] Unable to update cni config: No networks found in /etc/cni/net.d Oct 5 03:49:05 server kubelet[70917]: E1005 03:49:05.970886 70917 kubelet.go:2167] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized |
貌似是网络的问题，根据master创建后给出的提示，我们应该把网络配置好，目的是让pod之间能够互相通讯。
配置网络
根据如下提示，我们可以通过kubectl apply命令配置网络。
|
1 2 3 4 |
You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ |
我们使用的是flannel的网络
|
1 2 3 4 5 6 7 8 9 10 11 |
k8s@server:~$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/flannel created serviceaccount/flannel created configmap/kube-flannel-cfg created daemonset.extensions/kube-flannel-ds-amd64 created daemonset.extensions/kube-flannel-ds-arm64 created daemonset.extensions/kube-flannel-ds-arm created daemonset.extensions/kube-flannel-ds-ppc64le created daemonset.extensions/kube-flannel-ds-s390x created |
注意：K8S的网络配置信息保存在/etc/cni/net.d目录下面，这个目录默认是不存在的，K8S也不会帮你创建，因此执行kubectl apply之前必须先把它创建好，否则网络会配置失败
命令执行后K8S会从网络拉取额外的pod，视乎网络的性能，可能需要耐心等待
配置完毕，目录下会多了个文件
|
1 2 3 |
k8s@server:~$ ls /etc/cni/net.d/ 10-flannel.conflist |
现在再查看master的状态
|
1 2 3 4 |
k8s@server:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION server Ready master 30m v1.12.0 |
状态已经变为Ready，至此master配置完毕。
添加其他节点
master配置过程中已经给出了加入节点的命令，其他节点只需要执行此命令即可
注意：非master节点同样需要关闭swap分区。
|
1 2 3 |
kubeadm join 192.168.11.161:6443 --token lh3ffu.9yr2wml1ahetaz7l \ --discovery-token-ca-cert-hash sha256:37cdfea7ff4e2da10571a02610d49d4de3c37770a4e720c6a84f124302a18551 |
分别在两个非master节点执行上面的命令
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
root@ubuntu-host1:~# kubeadm join 192.168.11.161:6443 --token lh3ffu.9yr2wml1ahetaz7l --discovery-token-ca-cert-hash sha256:37cdfea7ff4e2da10571a02610d49d4de3c37770a4e720c6a84f124302a18551 [preflight] running pre-flight checks [WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh] or no builtin kernel ipvs support: map[ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{}] you can solve this problem with following methods: 1. Run 'modprobe -- ' to load missing kernel modules; 2. Provide the missing builtin kernel ipvs support [discovery] Trying to connect to API Server "192.168.11.161:6443" [discovery] Created cluster-info discovery client, requesting info from "https://192.168.11.161:6443" [discovery] Requesting info from "https://192.168.11.161:6443" again to validate TLS against the pinned public key [discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.11.161:6443" [discovery] Successfully established connection with API Server "192.168.11.161:6443" [kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.12" ConfigMap in the kube-system namespace [kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [preflight] Activating the kubelet service [tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap... [patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "ubuntu-host1" as an annotation This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the master to see this node join the cluster. |
耐心等待一段时间，因为需要准备一些pod
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
k8s@server:~$ kubectl get pod --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-576cbf47c7-4ldh6 1/1 Running 0 37m kube-system coredns-576cbf47c7-c4p6n 1/1 Running 0 37m kube-system etcd-server 1/1 Running 0 37m kube-system kube-apiserver-server 1/1 Running 0 37m kube-system kube-controller-manager-server 1/1 Running 0 37m kube-system kube-flannel-ds-amd64-bztzt 0/1 Init:0/1 0 3m57s kube-system kube-flannel-ds-amd64-kcwwz 0/1 Init:0/1 0 2m57s kube-system kube-flannel-ds-amd64-nt6dp 1/1 Running 0 15m kube-system kube-proxy-snvvd 1/1 Running 0 2m57s kube-system kube-proxy-tv6jp 1/1 Running 0 37m kube-system kube-proxy-vjl6f 1/1 Running 0 3m57s kube-system kube-scheduler-server 1/1 Running 0 37m |
查看pod的拉取日志
|
1 2 3 4 5 6 7 8 |
k8s@server:~$ kubectl describe pod kube-flannel-ds-amd64-bztzt --namespace=kube-system ...... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 4m58s default-scheduler Successfully assigned kube-system/kube-flannel-ds-amd64-bztzt to ubuntu-host1 Normal Pulling <invalid> kubelet, ubuntu-host1 pulling image "quay.io/coreos/flannel:v0.10.0-amd64" |
拉取过程中可能会出错：net/http: TLS handshake timeout， 原因是墙的问题，可以让docker daemon走代理，配置如下：
|
1 2 3 4 5 6 7 |
$ sudo mkdir -p /etc/systemd/system/docker.service.d $ sudo vim/etc/systemd/system/docker.service.d/http-proxy.conf [Service] Environment="HTTP_PROXY=http://192.168.16.2:1080/" $ sudo systemctl daemon-reload $ sudo systemctl restart docker |
前提是这个代理可以访问被墙的网站，完了之后手动在非master节点拉取这个image即可。
|
1 2 |
root@ubuntu-host2:~# docker pull quay.io/coreos/flannel:v0.10.0-amd64 |
最后所有节点状态为Ready
|
1 2 3 4 5 6 |
k8s@server:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION server Ready master 66m v1.12.0 ubuntu-host1 Ready <none> 32m v1.12.0 ubuntu-host2 Ready <none> 31m v1.12.0 |
转载请注明：Pure nonsense » Kubernetes环境搭建
