I really like Kubernetes; I\u2019ve been following almost since its inception 5 years ago and used it successfully in the past 3+ years in several projects. It isn\u2019t without challenges (esp. around managing state) but it definitely getting better with each release. Moving to a new company, it is no wonder I introduced Kubernetes into our architecture from the get-go.<\/p>\n\n\n\n
One thing different in this project though, is that I am now running Kubernetes in the cloud (AWS for now), whereas in the last two times the target use was on-prem. Using Kubernetes in the cloud alleviate some of the pains we had to deal with self-hosting it, like installation, control plane availability, etc. but it also introduces some new challenges \u2013 one such challenge is around security.<\/p>\n\n\n\n
There are many ways to protect the inter-pod\/service communications inside Kubernetes (maybe I\u2019ll dedicate another post for that) \u2013 the problem here is different, it is controlling the security from the different pods and other AWS assets (like RDS, S3, other EKS clusters, etc.).<\/p>\n\n\n\n
One way to handle security in AWS is to associate an AWS role with an instance. That works well in the \u201cclassic\u201d AWS setup since different instances (or groups of instances) host different services. This is not the case when using Kubernetes, now we have multiple types of service (internal and external) running on the same node \u2013 if the node has the maximal security we\u2019re not only violating the \u201cleast privileges principle\u201d for our own services, we\u2019re probably also exposing our AWS resources to 3rd party pods we\u2019re running on the same nodes.<\/p>\n\n\n\n
It seems that one possible solution is to set the node permissions to something minimal and to AWS key pairs for each service. This has 2 problems \u2013 one that the nodes need some privileges to be part of the Kubernetes cluster also it is a major headache to store and distribute the key pairs in a secure manner (e.g. making sure they don\u2019t end up hard-coded in source code; making sure they are not propagated to pods as plain text; etc)<\/p>\n\n\n\n
Luckily, there\u2019s a better approach \u2013 that brings the IAM based approach from regular ec2 instances to the pod level. There are a couple of tools, that I know of, that support this, namely kube2iam<\/a> and kiam<\/a> (you can read a nice comparison of the two here<\/a>)<\/p>\n\n\n\n Though they work a little differently, they are both based on the same approach:<\/p>\n\n\n\n and presto \u2013 your pod only has the privileges it was configured with<\/p>\n\n\n\n To get this magic going you need to do 3 things:<\/p>\n\n\n\n The biggest problem for me was setting the permissions right. to do that you<\/p>\n\n\n\n 1. add a new policy with sts:assume permission<\/p>\n\n\n\n Note that you can (and should) limit the resources that services can assume by specifying a resource prefix<\/p>\n\n\n\n 2. find what role is used by the worker nodes in your cluster and add to it the policy from step 1<\/p>\n\n\n\n 3. for each role that you define and want pods to use you also need to add a trust relationship that allows the worker nodes\u2019 role to assume it<\/p>\n\n\n\n The kube2iam site has instructions on installing it \u2013but I found it was easier to install it using helm:<\/p>\n\n\n\n where the config file is:<\/p>\n\n\n\nset up permissions<\/h3>\n\n\n\n
\n\n
\n \n \n {<\/code><\/div>\n <\/code>\"Version\"<\/code>: <\/code>\"2012-10-17\"<\/code>,<\/code><\/div>\n <\/code>\"Statement\"<\/code>: [<\/code><\/div>\n <\/code>{<\/code><\/div>\n <\/code>\"Sid\"<\/code>: <\/code>\"\"<\/code>,<\/code><\/div>\n <\/code>\"Effect\"<\/code>: <\/code>\"Allow\"<\/code>,<\/code><\/div>\n <\/code>\"Action\"<\/code>: <\/code>\"sts:AssumeRole\"<\/code>,<\/code><\/div>\n <\/code>\"Resource\"<\/code>: <\/code>\"arn:aws:iam::your-account-id:role\/prefix*\"<\/code><\/div>\n <\/code>}<\/code><\/div>\n <\/code>]<\/code><\/div>\n}<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n\n\n\n\n\n
\n \n \n {<\/code><\/div>\n <\/code>\"Version\"<\/code>: <\/code>\"2012-10-17\"<\/code>,<\/code><\/div>\n <\/code>\"Statement\"<\/code>: [<\/code><\/div>\n <\/code>{<\/code><\/div>\n <\/code>\"Effect\"<\/code>: <\/code>\"Allow\"<\/code>,<\/code><\/div>\n <\/code>\"Principal\"<\/code>: {<\/code><\/div>\n <\/code>\"Service\"<\/code>: <\/code>\"ec2.amazonaws.com\"<\/code><\/div>\n <\/code>},<\/code><\/div>\n <\/code>\"Action\"<\/code>: <\/code>\"sts:AssumeRole\"<\/code><\/div>\n <\/code>},<\/code><\/div>\n <\/code>{<\/code><\/div>\n <\/code>\"Sid\"<\/code>: <\/code>\"\"<\/code>,<\/code><\/div>\n <\/code>\"Effect\"<\/code>: <\/code>\"Allow\"<\/code>,<\/code><\/div>\n <\/code>\"Principal\"<\/code>: {<\/code><\/div>\n <\/code>\"AWS\"<\/code>: \"arn:aws:iam::your-account-id:role\/worker-node-role<\/code><\/div>\n <\/code>},<\/code><\/div>\n <\/code>\"Action\"<\/code>: <\/code>\"sts:AssumeRole\"<\/code><\/div>\n <\/code>}<\/code><\/div>\n <\/code>]<\/code><\/div>\n <\/code>}<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n\n\n\nInstalling kube2iam<\/h3>\n\n\n\n
\n\n
\n \n \n helm install stable\/kube2iam --name dev-kube2iam --<\/code>namespace<\/code> kube-<\/code>system<\/code> -f .\/kube2iam.config.yaml<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n\n\n\n\n\n
\n \n \n aws:<\/code><\/div>\n <\/code>region: <\/code>\"your-aws-region\"<\/code><\/div>\nextraArgs:<\/code><\/div>\n <\/code>auto<\/code>-discover-base-arn: <\/code>true<\/code><\/div>\n <\/code>auto<\/code>-discover-<\/code>default<\/code>-role: <\/code>true<\/code><\/div>\n# Won't work with Calico<\/code><\/div>\nhost:<\/code><\/div>\n <\/code>iptables: <\/code>true<\/code><\/div>\n <\/code>interface: eni+<\/code><\/div>\nrbac:<\/code><\/div>\n