1.

CORE-CONCEPT KNOWLEDGE

No Error - ONLY connected by kube-apiserver Debug step

1 etcd server -- validate
Has 2 ports type:
request
2 kube-apiserver - [Link]:2379:
retrived / update for kube-apiserver
data communicate (1)
=> ETCD servers
pod qua
(only node #)interact with ETCD)
componet
3 kube-controller-manager
- Step2: calculate & find the most suitable node
4 kube-scheduler
- Step3: send signal to KUBELET for pod creation
5 kubelet Contact points between worker nodes & control plan
6 kube-proxy - pods communication channel in same nodes / across nodes (NAT)

ReplicationController
7 ReplicaSet
Deployment

- can access to service clusterIP by ssh to worker nodes & curl cluster IP
8 Service Ex: kubectl get service | grep uac => clusterIP: [Link]:80
9 curl [Link] => {OK}
10
11
 KNOWLEDGE

Debug step
kube-apiserver
(1) & (2)
Can
=> view be deployed as:
in /etc/kubernetes/manifests/[Link]
kube-apiserver communicate (1)
a => ETCD servers - Pod
Can bein k8s
deployed as:
act with ETCD) -- systemd
Pod services
Can bein k8s
deployed as:
nd the most suitable node
-- Pod
systemd services
in k8s
o KUBELET for pod creation
en worker nodes & control plan - -deployed
systemd only
services
as systemd services
n channel in same nodes / across nodes (NAT) - deployed as daemonSet

Main difference (RC vs RS)

=> "Selector" to specify which Pod is included in RS

Useful when:
- Many pod with "labels" exists before of RS creation
=> create RS to include them all
ce clusterIP by ssh to worker nodes & curl cluster IP
Reason: because of docker networking
e | grep uac => clusterIP: [Link]:80
- docker0 as NAT ("veth" connect between docker virtual network & hos
5.252/health => {OK}
netes/manifests/[Link]

ystemd services
nSet

vs RS)
ify which Pod is included in RS

ls" exists before of RS creation

de them all
docker networking
eth" connect between docker virtual network & host network)
 [Link] KNOWLEDGE

No Error /etc/kubernetes/manifests Debug

2 dynamicPod vs staticPod => all manifests in /etc/kubernetes/manife
[Link]/hostname
3 assign Pod to nodes (part1) [Link]/zone
[Link]/instance-type
4
- PreferNoSchedule: 50% sure for NOT assign new pod 3) Affinity & anti-affinity:
- Taints: used for NODES
- Toleration: use for PODs, only 80% guaran
- Affinity: use for PODs, 100% guaran
TAINTS / TOLERATION vs AFFINITY
5
Differentiate ??? Example for (*):
- Nodes: taints by "blue / red / green"
- Pods: tolerate also by "blue/red/green".
But there's case, pod still go to "OTHER"

5 Requests / Limits -- ifDynamic (with Scheduler)

specify "REQUESTS" = 64Mi, not limit
6 How many way to deploy & assign pod => use scheduler to determine worker nod
How many methods to schedule pod to nodes
- Manually -> using "nodeName" in pod m
7 kubectl get events - events for scheduling pod into workers
8 select pod based on "label" kp
B) --selector env=dev
If same <my-2nd-scheduler> on 2 masters
9 deploy "additional custom scheduler" on same cluster => use options:
--scheduler-name=<my-2nd-scheduler>
NG KNOWLEDGE

kubernetes/manifests Debug step

all manifests in /etc/kubernetes/manifests are created when kubelet starts
[Link]/hostname
[Link]/zone
[Link]/instance-type
- matchExpressions: (A)
ffinity & anti-affinity:
- key: ....
nts: used for NODES
eration: use for PODs, only 80% guarantee pods go to correct node (*)
nity: use for PODs, 100% guarantee pods go to correct node (use with taints)

mple for (*):

des: taints by "blue / red / green"
ds: tolerate also by "blue/red/green".
there's case, pod still go to "OTHER"
- default:
amic (with Scheduler) : start
notbylimit
calling to API Server
ecify "REQUESTS" = 64Mi, spec: memory: 512Mi
use scheduler to determine worker node
containers:
nually -> using "nodeName" in pod manifest yaml
nts for scheduling pod into workers
selector env=dev
same <my-2nd-scheduler> on 2 masters, which will becomes main scheduler ??? name: mypod
se options: spec:
eduler-name=<my-2nd-scheduler> schedulerName: my-2nd-scheduler
 image: nginx:1.7.9
ons: (A)
ports:

2nd-scheduler
 LOGGING & MONITORING KNOWLEDGE

No Error Debug step

cAdvisor - containerAdvisor verify:

(subcomponent of kubelet) --------------
k top nodes
=> collect & report metrics of containers
to "metrics Servers" k top pods
 APPLICATION LIFE CYCLE

No Error Debug step

- Recreate: delete ALL, then start ALL
1 deployment "StrategyType"
-# Rolling update: update one-by-one
Get status
2 rolling update
kubectl rollout status [Link]/nginx-
3 rollback
4 how many pods can be down at rollingUpd- use RollingUpdateStrategy: 25%maxUnavailable
pod specs: command & args in Pod Spec
4 command: overwrite ENTRYPOINT
args: overwrite CMD Secret / ConfigMap
echo -n "string_secret" | base64
5 create secret base64 -=>env (get each key "\n"
seperately)
remove newline character
6 Inject into Pods - envFrom (get all key at once)
- mountVolume
FE CYCLE

ug step Note
hen start ALL
one-by-one
# Scaling deployment
[Link]/nginx- kubectl scale [Link]/nginx-
egy: 25%maxUnavailable
n Pod Spec

gMap
base64
rately)
haracter
once)
 [Link] MAINTENANCE KNOWLEDGE

No Error Debug step

"--pod-eviction-timeout=5m0s" /
2 kpod eviction
drain time
node01 node01
OR
3 k cordon node01 -- master
markednodes"unscheduling"
are "tained"
4 kwhy
uncordon node01 - give the node back
master nodes can not be assigned po only pod with "toleration" to normal
couldstate (can beassign
be forceful
to master nodes cluster process
kubelet [Link]
/ kube-proxy: X-2
5 versions rule for cluster component kubectl:upgrade
Step2: X-1 => X+1workers
6 upgrade process + Options2A: all workers at a time => down time (faster)
kubeadm upgrade apply <version> -+kubeadm
Options2B:upgrade
rollingappy 1.12.0
node-by-node => no down time (slower)
7
--------------------------------------------------------------------
[Link] & Restore Method (ETCD)
1. Query from API server & importer later on
8 Back up cluster
2. Backup & restore ETCD data
9
/etc/kubernetes/pki/etcd/[Link] [Link] -> cert file for CA whole etcd cluster
10
/etc/kubernetes/pki/etcd/[Link] [Link] -> cert file for server of etcd cluster
--key="/etc/kubernetes/pki/etcd/[Link]" \
11 NOTE: should practice again
--cert="/etc/kubernetes/pki/etcd/[Link]"
--initial-cluster-token etcd-cluster-1 \ \
12 [Link]
Restore ETCD data
--initial-advertise-peer-urls [Link] \
CE KNOWLEDGE

ug step Note
=5m0s" /

ed"
ormal
n" couldstate (can beassign
be forceful
2r process

s at a time => down time (faster)

yde-by-node
1.12.0 => no down timeWorker (slower)
node (only kubelet)
-----------------------------------
ethod (ETCD) - kubectl drain node01
& importer later on
D data
CA whole etcd cluster
server of etcd cluster
netes/pki/etcd/[Link]" \ -> using etcdctl API version "3"
netes/pki/etcd/[Link]"
cd-cluster-1 \ \ ->
-> remember to API
using etcdctl use version "3"
urls [Link] \
 SECURITY KNOWLEDGE

=path_to_csvfile" in [Link] Authentication & Certificates

- RBAC (Role based...)
1 / Certificates (TLS) - ABAC (Attribute based...)
/ LDAP - Node Authorization

- Private key (có chữ "key" trong filename)

2 TLS
- Public key (others: pem, crt)

=> /etc/kubernetes/pki/{[Link], [Link]}

3 2 CA (certificate Authority)
- For etcd cluster
"Certificate + key" (.crt + .key) cho từng:
1) services (client) / end-user
- admin
- kube-scheduler
- kube-controller-manager
- kube-proxy
=> talk to kube-apiserver

4 2) đầu server / client

Below talks to "Server"
- apiserver-kubelet (client => kubelet)
- apiserver-etcd (client => etcd)

Below server get requests from "client"

- kubelet (server)
- etcd cluster (server)
- api-server (server) (/etc/k8s/pki/{[Link],[Link]}
5 Quy trình tạo cert cho new user
- create role + rolebinding with specific RBAC (only get 2. pod,
Tạo "private
...) key" cho new user
- verify permission "baohuynh-test" with "can-i" plugin 3. Tạo "certificate signing request"
- expire day (*)
6 Có thể có 2 CA cấp)
(certificate Authorization): openssl x509 -noout -text -in [Link]
- issuer (người
7 -Debug
[1] CAincho K8S
case service (kube-apiserver,
kube-apiserver failed kube-scheduler,...)
[1]
=> -check
/etc/kubernetes/pki/
wrong path_cert {[Link], [Link]}
8 - [2] CA cho etcd cluster [2] - /etc/kubernetes/pki/etcd/
B) " docker ps -a | grep kube-api {[Link], [Link]} logs
" =>"docker
Example test: [Link] <..>"
kube cluster
9
Thay vì phải tự tạo file .crt = cách lấy 2 file {[Link], [Link]}
3/As an admin, Appove/deny pending csr
- Role / Clusterrole RBAC Authorization
10 RBAC - Rolebinding / Clusterrolebinding
(for user/group/serviceAccount)

ResourceNames (1) giới hạn quyền, chỉ truy cập vào pod tên "BaoHuyn

NOTICE
 list all of apiGroups & related resources
=> needed "config" info for role / clusterrole
NOTICE
+ Pod, deployment, svc, secret,... => apiGroups: ["
kubectl api-resources (2) + persistentvolumes/... => apiGroups:
["[Link]"]

(check by "kubectl api-resrouce | grep <keywrord>

11 Inspect "authorization mode" -> view config in manifest of API-server
Aggregated ClusterRoles
12
Ex:

Image security
1. Create secret (stores docker login credentials)
--> see next cell
2. Using imagePullSecrets (built-in) in .yaml
apiVersion: v1
kind: Pod
Pull from private repository: ................
13
spec:
containers:
- name: private-reg-container
image: <your-private-image>
imagePullSecrets:
- name: private-reg-cred
Security Context
runAsGroup: 3000 runAsUser: 1000
- capabilities => some priviledges in Linux runAsGroup: 3000
- from Network Policies
user: alice
- to
podSelector:
+ ipBlock
ntication
..) & Certificates
ed...)

"key" trong filename)

pem, crt)

pki/{[Link], [Link]}

[Link]}
-subj "/O=$GROUP/CN=$USER"
ho new user
4) openssl x509 -req -days 3650 -sha256 -in $CLIENTCERTCSR \
ning request"
ut -text -in [Link]
pki/
_cert{[Link], [Link]}
pki/etcd/
ep kube-api{[Link], [Link]} logs
" =>"docker C) Cert expires
=> regenerate from .csr + .key
spec:
ve/deny pending csr request: $(cat [Link] | base64 | tr -d '\n')
BAC Authorization Impersonate & check access
rrolebinding - kubectl auth can-i <user-name> get pods
ceAccount) -> yes / no

uy cập vào pod tên "BaoHuynh" apiVersion: [Link]/v1

kind: ClusterRole
metadata:
name: storage-admin
rules:
- apiGroups: ["[Link]"] <= (1)
resources: ["storageclasses", "persistentvolumes"]
resourceNames: ["BaoHuynh"] <= (2)
verbs: ["*"]
 apiVersion: [Link]/v1
kind: ClusterRole
related resources metadata:
fo for role / clusterrole name: storage-admin
rules:
vc, secret,... => apiGroups: [""] - apiGroups: ["[Link]"] <= (1)
.. => apiGroups: resources: ["storageclasses", "persistentvolumes"]
resourceNames: ["BaoHuynh"] <= (2)
verbs: ["*"]
i-resrouce | grep <keywrord>)
ifest of API-server name: monitoring
aggregationRule:
clusterRoleSelectors:
Image security
es docker login credentials)

rets (built-in) in .yaml kubectl create secret docker-registry \

private-reg-cred \
--docker-username=dock_user \
--docker-password=dock_password \
--docker-email=dock_user@[Link] \
--docker-server=[Link]
container
te-image>
cred
Security Context

Network Policies matchLabels:

role: client
- ipBlock:
 labels:
[Link]/aggregate-to-monitoring: "true"
# These rules will be added to the "monitoring" role.
 STORAGE

No Error delete) Debug step

1 emptyDir - Can be mounted at different path of multi-container
EBS
inside/ AzureDisk
a Pod <----> PV <--------> PVC <-------> Pod Volume Declare
3 -Flow
PVCVolume Usage - NFS (ngag hàg với Container trong
2 - Mount -directly
....... into Pod
Declare
  Volume
5 + Create a volume for PVC / Direct hostPath
Volume Mount (inside Pod) storage
4 PV reclaiming policies + Create MountPath for using that volume
deleting will make PVC in state Root cause:on cloud provider still exist EBS, Azure
PVC is "in-use" by pod
6 emtyDir:
"Terminating" forever => couldatnot "terminated"
7 emtyDir vs hostPath - Define "Pod"
8 - Exist during Pod life-cycle
9
 Note

------> Pod Volume Declare <----> Pod VolumeMount (with Path)

gag hàg với Container trong yaml)
apiVersion: v1 task-pv-claim
claimName: accessModes:
kind: PersistentVolume
container:
- Delete: disappear with PVC
- Recycle: clear old data + available for
hostPath:
- Define at "PV"
- Exists outside pod's life-cycle
 claimName: task-pv-claim
containers:
- name: task-pv-container
 NETWORKING KNOWLEDGE
No Error Debug step
- standards cho việc thiết lập các
1
route/forward-traffic between host & - kube-apiserver: public ([Link]:6443)
2 Cluster networking exposure - kubelet: public ([Link]:10250)
- etcd: [Link] / Overlay
public ([Link]:2379)
network ( weave )
inpsect which network plugin is used => --cni-bin-dir=/opt/cni/bin \ => (list plugins carried by weave)
4 weave plugin (support CNI) =>
by kubelet - (assign IP & create NAT table) for (configuration
--cni-conf-dir=/etc/cni/net.d \ => traffic for weave)
5 daemonset:
docker0 vs weave ([Link]/12) internal IP -> external IP
6 pod get IP from weave subnet
PODs
([Link]/12)
8
- Define by weave add-on (daemonset)
[Link]-proxy (Service network)
- accessible cluster wide (from any places in
10 Service ClusterIP
cluster)
kube-apiserver:
11 Where defines service IP ranges
kube-proxy --service-cluster-ip-range=[Link]/12
- represent the "Service" + convey traffic
12 => define IP of Pod & Service
from/to "service"
13 CNI (weave) vs kube-proxy
2/ kube-proxy [Link] in K8S
curl [Link]
14 DNS record hold by coredns
- Full-qualified name for Pod (MUST)

- configmap - coredns => main configure for

CoreDNS
15
- daemonset + service - coredns (use above
cm as Config File)
where to config root domain name
server ([Link]) ???

$ cat /var/lib/kubelet/[Link]:
..................
clusterDNS:
16
- [Link] => IP for coreDNS service
clusterDomain: [Link]
----------
17 examine root domain for each pod search [Link] \
+ subdomain: name of svc name: default-subdomain
[Link] \
18
spec:
- ClusterFirst: match "*.[Link]" then using
19 Pod DNS Policies
DNS by coredns, other will use DNS resolve
[Link]
 Service ( Ingress ) (nodePort / LB)
||
V
Services (wear svc / video svc)
||
20 Ingress overview
V
Backend Pods

- Ingress Controller
- Ingress Resource (/trogiup, /vuitrangram)

Host configure on the ingress-resource

HOST:
EDGE

- kube-scheduler: localhost:10251
- kube-controller-manager: localhost:10252
weave )
list plugins carried by weave)
configuration for weave)
+ role + clusterrole
+ rolebinding + clusterrolebinding
{ "dst": "[Link]" }
]
work) }

- without kube-proxy, no routing table => packet don't

configmap - coredns
=> mount to /etc/coredns/Corefile in Pod deploy network policies
----
.:53 {
errors health
kubernetes [Link] [Link] [Link]
{
pods insecure
upstream fallthrough [Link] [Link]
}
prometheus :9153
proxy . /etc/[Link]
cache 30
reload
}
For every pod started by Kubelet, it will inherit configure
from this:
/var/lib/kubelet/[Link]

lookup svc IP in different namespaces:

kubectl exec -it myweb-xyz nslookup [Link]
>hostname: busybox-1
[Link]
subdomain: default-subdomain nameservers:
- None: ignore k8s cluster DNS configure, use it owns - [Link]
"dnsConfig" on pod specs
[Link] searches:
Architect

Ingress Controller architecture:

-----------------------------------------
- configmap:
+ SSL, logpath, header,.....

- Ingress controller (deployment):

+ container: ....
+ args: --configmap=$(POD_NS)/nginx-configmap
+ env: ....
+ ports: http(80) & https (443)

- Service
+ serviceType: nodePort / LB

- Authentication:
-+ ServiceAccount
with + clusterrolebinding
"Host": only for that domain
(ex: [Link])
=> [Link]/v1beta1 => extensions/v1beta1 check by "kubectl api-resources"
i-resources"
 TROUBLESHOOTING KNOWLEDGE

No Error Debug
- check application configure (DB_user, step
DB_pass,...)
1 Application failure + kube-replication-controller-manager
2 ControlPlan failure Verify application working:
+ kube-scheduler
+ kube-scheduler
3 node failure
4 networking failure ???
 NOWLEDGE

Debug
onfigure (DB_user, step
DB_pass,...)
ontroller-manager
king:
 OTHER KNOWLEDGE

No Useful when (wildcart): Error Debug step

kn k8s-master-v3-77 -o jsonpath
1 - getting all of nodes's status
-json
getting - if [ ] :=> list / array
2 withcurrent restart
root format (.) count....
new line {"\n"} -Ex:
if {} :=> dictionary
3
tab {"\t"} kn -o jsonpath='{.items[*].[Link]}

--custom- Ex: sort pv by their capacity with custom column fir

4 columns=HEADER1:.[Link],HEADER2:.spe k get pv -o custom-
[Link] columns=NAME:.[Link],CAPACITY:.spec
by='{.[Link]}' > pv-and-capacity-sor
5 --sort-by='{[Link]}'
**** IF conditon **** NOTE: --custom-columns + --sort-by DONT use .ite
6 k config view --kubeconfig=/root/my-kube-config -o
Ex: select the context (name) for user "aws-user" by
- jsonpath='{}' (@.[Link] == "aws-user")]
='{.items[*]}'
Ex: kn -o jsonpath='{range .items
='{range .items[*]}'
.[Link][*]}{.type}{.status
='{range .items[*]}{.[Link]}:'
='{range .items[*]}{.[Link]}:{.[Link][*]}'
='{range .items[*]}{.[Link]}:{range .[Link][*]}{.effect}'
='{range .items[*]}{.[Link]}:{range .[Link][*]}{.effect}{.key}'
='{range .items[*]}{"\n"}{.[Link]}:{range .[Link][*]}{.effect}{.key}'
E

Debug step
-o jsonpath='{.[Link]}' (syntax like awk)

s[*].[Link]}{"\n"}{.items[*].[Link]}'
apacity with custom column first

[Link],CAPACITY:.[Link] --sort-
torage}' > [Link]

mns + --sort-by DONT use .item[*] in query

onfig=/root/my-kube-config -o jsonpath='{.contexts[?
kn -o jsonpath='{range
aws-user")].name}' .items[*]}{"\n"}{.[Link]}:{range .[Link][*]}--
> /opt/outputs/aws-context-name
{.effect}'
kn -o jsonpath='{range .items[*]}{"\n"}{.[Link]}:{range .[Link][*]}
kn -o jsonpath='{range .items[*]}{"\n"}{.[Link]}:{.[Link][*].effect}:{range
{.type}:{.status}**'
[Link][*]}{.type}{.status}--'
 Kubernetes The Hard Way

No Error option: Debug step

1 - High availability kube-controller-manager
kube-scheduler ETCD HA cluster
Leader election - Raft algorithm (first node appear when
2
mechanism create cluster)
ALWAYS choose "ODD number"

=> 1. For Quorum with "node down" Tolerance & g

using Quorum = N/2 + 1
=> 2. In case of network split into 2 separate ones
3 Faiure Tolerance -> If 3 nodes, 2 write ok -> successful
suppose cluster got 6 nodes
write to cluster
each splitted cluster have 3 nodes
(accept 1 down)
-> don't have enough QUORUM (4 nodes) for s
-> corrupted
(see below example)

5
6
7
8
9
se "ODD number"

um with "node down" Tolerance & get less VMs setup

f network split into 2 separate ones

uster got 6 nodes
d cluster have 3 nodes
e enough QUORUM (4 nodes) for successful write
d
example)
 OTHER KNOWLEDGE

No Error Ex: Debug step

1 kubectl proxy POD_NAME=$(kubectl
- kubectl get pods -o json | jq
run --generator=deployment/v1beta1 -r '.items[].m
nginx --image=
2 - Using kubectl to create pod curl [Link]
--dry-run -o yaml
k run nginx --image=nginx --restart=Never
3 run pod without create Deployment
k run nginx --image=nginx --restart=Never -o yaml > [Link]
kubectl config view --kubeconfig=/root/my-kube-config
EDGE

Debug step
ubectl get pods -o json | jq
tor=deployment/v1beta1 -r '.items[].[Link]'
nginx )
--image=nginx --replicas=4
st:8001/api/v1/namespaces/default/pods/$POD_NAME/proxy/
nginx --restart=Never
nginx --restart=Never -o yaml > [Link]
kubeconfig=/root/my-kube-config