{"id":3065,"date":"2017-09-12T17:15:23","date_gmt":"2017-09-12T14:15:23","guid":{"rendered":"https:\/\/www.systemcodegeeks.com\/?p=3065"},"modified":"2017-09-12T12:16:53","modified_gmt":"2017-09-12T09:16:53","slug":"kubernetes-1-7-centos-nailed","status":"publish","type":"post","link":"https:\/\/www.systemcodegeeks.com\/devops\/kubernetes-1-7-centos-nailed\/","title":{"rendered":"Kubernetes 1.7 on CentOS: This is How We Nailed It!"},"content":{"rendered":"

Please Note:<\/b> If you are looking for a comprehensive, step-by-step guide on installing Kubernetes on CentOS, check out this guide<\/a> (composed by one of my colleagues) instead.<\/i><\/p>\n

Caution:<\/b> This blog was originally composed several months ago, when the K8s release cycle was at v1.7.0-alpha.3<\/code>. As always, K8s has zoomed past us<\/a>, hence most of the content would be downright obsolete by now!<\/i><\/p>\n

If you are a fan of Kubernetes (K8s)<\/a> and the ton of amazing features it has in store for developers, you would surely agree with me on the fact that getting a local K8s setup up and running<\/a> is one of the most fundamental steps of learning K8s.<\/p>\n

Since we have been waiting for quite some time to re-spin our Integration Platform (IPS)<\/a> product on top of a new K8s version, we decided it was high time to get a brand new cluster up and running with 1.7.0 (which is still in beta as of this writing, but would become stable by the time we migrate our own code on top of it). Due to some stability requirements we wanted a setup running a RedHat-compatible base OS, so we decided to go ahead with CentOS.<\/p>\n

While we could have always have followed one of the pre-compiled guides<\/a> from K8s docs, such as this<\/a>, we wanted to be a bit more adventurous and get things done by way of the K8s source; besides, we wanted to be able to upgrade our set-up when required, for which the best way was to properly understand how the K8s authors have nailed it in the original source.<\/p>\n

First of all, we got hold of a clone of the K8s source<\/a> at the branch master<\/code> branch (as we needed the latest code base) and tag v1.7.0-alpha.3<\/code> (since we were looking for that specific release).<\/p>\n

Next we built the K8s binary artifacts using the make<\/code> command at sources root. (You can utilize the Docker-based cross compilation approach mentioned in the above guide as well, but note that it would download a 1.8+ GB cross compiler image and build artifacts for all platforms resulting in a 20+ GB total size.)<\/p>\n

We set up a VirtualBox<\/a> virtual machine running CentOS 7, and installed docker<\/code>, etcd<\/code> and flanneld<\/code> afresh. Docker was installed via yum<\/code> (sudo yum install docker<\/code>) and configured for use by our default, non-root user<\/a>. Others were installed via yum<\/code> as well, with versions etcd-3.1.3-1.el7.x86_64.rpm<\/code> and flannel-0.7.0-1.el7.x86_64.rpm<\/code>, respectively.<\/p>\n

We were hoping to clone the machine in order to obtain a multi-node set-up (without undertaking the overhead of installing the individual components on each machine separately). However, this seemingly complicated the whole process.<\/p>\n

The default K8s installer for CentOS (located at cluster\/centos<\/code><\/a>) by default downloads archives of K8s client\/server, etcd<\/code>, flanneld<\/code> and docker<\/code> binaries as part of the configuration process. Since we needed to install K8s binaries from our custom build instead, and didn’t want to download the other binaries at all (as they were already installed), we had to customize the corresponding build and deployment scripts to avoid downloading the artifacts and utilize the build artifacts instead, and to control the reconfiguration of etcd<\/code>, flanneld<\/code> and docker<\/code>.<\/p>\n

What follows is a breakdown of the changes we had to perform on top of the v1.7.0-alpha.3<\/code> tag to get things working with native (yum<\/code>-based) installations of etcd<\/code>, flanneld<\/code> and docker<\/code>, with some attempts to explain the rationale behind each change.<\/p>\n

Tip:<\/b><\/i> If you wish to utilize any of these patches, it should be possible to save them into a plaintext file (with or without the .patch<\/code> extension) and apply them on top of v1.7.0-alpha.3<\/code> tag of the K8s source using the git apply<\/code> command.<\/p>\n

Tip:<\/b><\/i> If you only have a subset of the above services (etcd<\/code>, flanneld<\/code> and docker<\/code>) already installed, patch out only the appropriate files\/sections.<\/p>\n

    \n
  1. Avoiding download of binary artifacts:\n
    diff --git a\/cluster\/centos\/build.sh b\/cluster\/centos\/build.sh\r\nindex 18bbe6f..09a3631 100755\r\n--- a\/cluster\/centos\/build.sh\r\n+++ b\/cluster\/centos\/build.sh\r\n@@ -42,19 +42,6 @@ function clean-up() {\r\n function download-releases() {\r\n   rm -rf ${RELEASES_DIR}\r\n   mkdir -p ${RELEASES_DIR}\r\n-\r\n-  echo \"Download flannel release v${FLANNEL_VERSION} ...\"\r\n-  curl -L ${FLANNEL_DOWNLOAD_URL} -o ${RELEASES_DIR}\/flannel.tar.gz\r\n-\r\n-  echo \"Download etcd release v${ETCD_VERSION} ...\"\r\n-  curl -L ${ETCD_DOWNLOAD_URL} -o ${RELEASES_DIR}\/etcd.tar.gz\r\n-\r\n-  echo \"Download kubernetes release v${K8S_VERSION} ...\"\r\n-  curl -L ${K8S_CLIENT_DOWNLOAD_URL} -o ${RELEASES_DIR}\/kubernetes-client-linux-amd64.tar.gz\r\n-  curl -L ${K8S_SERVER_DOWNLOAD_URL} -o ${RELEASES_DIR}\/kubernetes-server-linux-amd64.tar.gz\r\n-\r\n-  echo \"Download docker release v${DOCKER_VERSION} ...\"\r\n-  curl -L ${DOCKER_DOWNLOAD_URL} -o ${RELEASES_DIR}\/docker.tar.gz\r\n }\r\n \r\n function unpack-releases() {\r\n@@ -80,19 +67,12 @@ function unpack-releases() {\r\n   fi\r\n \r\n   # k8s\r\n-  if [[ -f ${RELEASES_DIR}\/kubernetes-client-linux-amd64.tar.gz ]] ; then\r\n-    tar xzf ${RELEASES_DIR}\/kubernetes-client-linux-amd64.tar.gz -C ${RELEASES_DIR}\r\n     cp ${RELEASES_DIR}\/kubernetes\/client\/bin\/kubectl ${BINARY_DIR}\r\n-  fi\r\n-\r\n-  if [[ -f ${RELEASES_DIR}\/kubernetes-server-linux-amd64.tar.gz ]] ; then\r\n-    tar xzf ${RELEASES_DIR}\/kubernetes-server-linux-amd64.tar.gz -C ${RELEASES_DIR}\r\n     cp ${RELEASES_DIR}\/kubernetes\/server\/bin\/kube-apiserver \\\r\n        ${RELEASES_DIR}\/kubernetes\/server\/bin\/kube-controller-manager \\\r\n        ${RELEASES_DIR}\/kubernetes\/server\/bin\/kube-scheduler ${BINARY_DIR}\/master\/bin\r\n     cp ${RELEASES_DIR}\/kubernetes\/server\/bin\/kubelet \\\r\n        ${RELEASES_DIR}\/kubernetes\/server\/bin\/kube-proxy ${BINARY_DIR}\/node\/bin\r\n-  fi\r\n \r\n   # docker\r\n   if [[ -f ${RELEASES_DIR}\/docker.tar.gz ]]; then<\/pre>\n<\/li>\n
  2. Avoiding verification of artifacts (since we no longer download them):\n
    diff --git a\/cluster\/kube-up.sh b\/cluster\/kube-up.sh\r\nindex 7877fb9..9e793ce 100755\r\n--- a\/cluster\/kube-up.sh\r\n+++ b\/cluster\/kube-up.sh\r\n@@ -40,8 +40,6 @@ fi\r\n \r\n echo \"... calling verify-prereqs\" >&2\r\n verify-prereqs\r\n-echo \"... calling verify-kube-binaries\" >&2\r\n-verify-kube-binaries\r\n \r\n if [[ \"${KUBE_STAGE_IMAGES:-}\" == \"true\" ]]; then\r\n   echo \"... staging images\" >&2<\/pre>\n<\/li>\n
  3. Coping for natively-installed etcd<\/code> and flanneld<\/code> binaries (which are located (rather symlinked) at \/usr\/bin<\/code> instead of \/opt\/kubernetes\/bin<\/code> where K8s expects them to be):\n
    diff --git a\/cluster\/centos\/master\/scripts\/etcd.sh b\/cluster\/centos\/master\/scripts\/etcd.sh\r\nindex aa73b57..6b575da 100755\r\n--- a\/cluster\/centos\/master\/scripts\/etcd.sh\r\n+++ b\/cluster\/centos\/master\/scripts\/etcd.sh\r\n@@ -74,7 +74,7 @@ Type=simple\r\n WorkingDirectory=${etcd_data_dir}\r\n EnvironmentFile=-\/opt\/kubernetes\/cfg\/etcd.conf\r\n # set GOMAXPROCS to number of processors\r\n-ExecStart=\/bin\/bash -c \"GOMAXPROCS=\\$(nproc) \/opt\/kubernetes\/bin\/etcd\"\r\n+ExecStart=\/bin\/bash -c \"GOMAXPROCS=\\$(nproc) \/usr\/bin\/etcd\"\r\n Type=notify\r\n \r\n [Install]\r\ndiff --git a\/cluster\/centos\/master\/scripts\/flannel.sh b\/cluster\/centos\/master\/scripts\/flannel.sh\r\nindex 092fcd8..5d9630d 100644\r\n--- a\/cluster\/centos\/master\/scripts\/flannel.sh\r\n+++ b\/cluster\/centos\/master\/scripts\/flannel.sh\r\n@@ -37,7 +37,7 @@ After=network.target\r\n \r\n [Service]\r\n EnvironmentFile=-\/opt\/kubernetes\/cfg\/flannel\r\n-ExecStart=\/opt\/kubernetes\/bin\/flanneld --ip-masq \\${FLANNEL_ETCD} \\${FLANNEL_ETCD_KEY} \\${FLANNEL_ETCD_CAFILE} \\${FLANNEL_ETCD_CERTFILE} \\${FLANNEL_ETCD_KEYFILE}\r\n+ExecStart=\/usr\/bin\/flanneld --ip-masq \\${FLANNEL_ETCD} \\${FLANNEL_ETCD_KEY} \\${FLANNEL_ETCD_CAFILE} \\${FLANNEL_ETCD_CERTFILE} \\${FLANNEL_ETCD_KEYFILE}\r\n \r\n Type=notify\r\n \r\n@@ -48,7 +48,7 @@ EOF\r\n # Store FLANNEL_NET to etcd.\r\n attempt=0\r\n while true; do\r\n-  \/opt\/kubernetes\/bin\/etcdctl --ca-file ${CA_FILE} --cert-file ${CERT_FILE} --key-file ${KEY_FILE} \\\r\n+  \/usr\/bin\/etcdctl --ca-file ${CA_FILE} --cert-file ${CERT_FILE} --key-file ${KEY_FILE} \\\r\n     --no-sync -C ${ETCD_SERVERS} \\\r\n     get \/coreos.com\/network\/config >\/dev\/null 2>&1\r\n   if [[ \"$?\" == 0 ]]; then\r\n@@ -59,7 +59,7 @@ while true; do\r\n       exit 2\r\n     fi\r\n \r\n-    \/opt\/kubernetes\/bin\/etcdctl --ca-file ${CA_FILE} --cert-file ${CERT_FILE} --key-file ${KEY_FILE} \\\r\n+    \/usr\/bin\/etcdctl --ca-file ${CA_FILE} --cert-file ${CERT_FILE} --key-file ${KEY_FILE} \\\r\n       --no-sync -C ${ETCD_SERVERS} \\\r\n       mk \/coreos.com\/network\/config \"{\\\"Network\\\":\\\"${FLANNEL_NET}\\\"}\" >\/dev\/null 2>&1\r\n     attempt=$((attempt+1))\r\ndiff --git a\/cluster\/centos\/node\/bin\/mk-docker-opts.sh b\/cluster\/centos\/node\/bin\/mk-docker-opts.sh\r\nindex 041d977..177ee9f 100755\r\n--- a\/cluster\/centos\/node\/bin\/mk-docker-opts.sh\r\n+++ b\/cluster\/centos\/node\/bin\/mk-docker-opts.sh\r\n@@ -69,7 +69,6 @@ done\r\n \r\n if [[ $indiv_opts = false ]] && [[ $combined_opts = false ]]; then\r\n   indiv_opts=true\r\n-  combined_opts=true\r\n fi\r\n \r\n if [[ -f \"$flannel_env\" ]]; then\r\ndiff --git a\/cluster\/centos\/node\/scripts\/docker.sh b\/cluster\/centos\/node\/scripts\/docker.sh\r\nindex 320446a..3f38f3e 100755\r\n--- a\/cluster\/centos\/node\/scripts\/docker.sh\r\n+++ b\/cluster\/centos\/node\/scripts\/docker.sh\r\n@@ -35,7 +35,7 @@ Type=notify\r\n EnvironmentFile=-\/run\/flannel\/docker\r\n EnvironmentFile=-\/opt\/kubernetes\/cfg\/docker\r\n WorkingDirectory=\/opt\/kubernetes\/bin\r\n-ExecStart=\/opt\/kubernetes\/bin\/dockerd \\$DOCKER_OPT_BIP \\$DOCKER_OPT_MTU \\$DOCKER_OPTS\r\n+ExecStart=\/usr\/bin\/dockerd \\$DOCKER_OPT_BIP \\$DOCKER_OPT_MTU \\$DOCKER_OPTS\r\n LimitNOFILE=1048576\r\n LimitNPROC=1048576\r\n \r\ndiff --git a\/cluster\/centos\/node\/scripts\/flannel.sh b\/cluster\/centos\/node\/scripts\/flannel.sh\r\nindex 2830dae..384788f 100755\r\n--- a\/cluster\/centos\/node\/scripts\/flannel.sh\r\n+++ b\/cluster\/centos\/node\/scripts\/flannel.sh\r\n@@ -39,7 +39,7 @@ Before=docker.service\r\n [Service]\r\n EnvironmentFile=-\/opt\/kubernetes\/cfg\/flannel\r\n ExecStartPre=\/opt\/kubernetes\/bin\/remove-docker0.sh\r\n-ExecStart=\/opt\/kubernetes\/bin\/flanneld --ip-masq \\${FLANNEL_ETCD} \\${FLANNEL_ETCD_KEY} \\${FLANNEL_ETCD_CAFILE} \\${FLANNEL_ETCD_CERTFILE} \\${FLANNEL_ETCD_KEYFILE}\r\n+ExecStart=\/usr\/bin\/flanneld --ip-masq \\${FLANNEL_ETCD} \\${FLANNEL_ETCD_KEY} \\${FLANNEL_ETCD_CAFILE} \\${FLANNEL_ETCD_CERTFILE} \\${FLANNEL_ETCD_KEYFILE}\r\n ExecStartPost=\/opt\/kubernetes\/bin\/mk-docker-opts.sh -d \/run\/flannel\/docker\r\n \r\n Type=notify\r\n@@ -52,7 +52,7 @@ EOF\r\n # Store FLANNEL_NET to etcd.\r\n attempt=0\r\n while true; do\r\n-  \/opt\/kubernetes\/bin\/etcdctl --ca-file ${CA_FILE} --cert-file ${CERT_FILE} --key-file ${KEY_FILE} \\\r\n+  \/usr\/bin\/etcdctl --ca-file ${CA_FILE} --cert-file ${CERT_FILE} --key-file ${KEY_FILE} \\\r\n     --no-sync -C ${ETCD_SERVERS} \\\r\n     get \/coreos.com\/network\/config >\/dev\/null 2>&1\r\n   if [[ \"$?\" == 0 ]]; then\r\n@@ -63,7 +63,7 @@ while true; do\r\n       exit 2\r\n     fi\r\n \r\n-    \/opt\/kubernetes\/bin\/etcdctl --ca-file ${CA_FILE} --cert-file ${CERT_FILE} --key-file ${KEY_FILE} \\\r\n+    \/usr\/bin\/etcdctl --ca-file ${CA_FILE} --cert-file ${CERT_FILE} --key-file ${KEY_FILE} \\\r\n       --no-sync -C ${ETCD_SERVERS} \\\r\n       mk \/coreos.com\/network\/config \"{\\\"Network\\\":\\\"${FLANNEL_NET}\\\"}\" >\/dev\/null 2>&1\r\n     attempt=$((attempt+1))<\/pre>\n<\/li>\n<\/ol>\n
    Once all fixes were in place, all we needed to do to get the ball rolling, was to run<\/p>\n
    MASTER=adrt@192.168.1.192 \\\r\nNODES=\"adrt@192.168.1.192 adrt@192.168.1.193 adrt@192.168.1.194\" \\\r\nDOCKER_OPTS=\"--insecure-registry=hub.adroitlogic.com:5000\" \\\r\nKUBERNETES_PROVIDER=centos \\\r\nCERT_GROUP=janaka \\\r\n.\/kube-up.sh<\/pre>\n
    within the cluster<\/code> directory of the K8s source.<\/p>\n