Etcd is a distributed, consistent and highly-available key value store used as the Kubernetes backing store for all cluster data, making it a core component of every K8s deployment.
\nDue to its central role etcd may contain sensitive information related to access of the deployed services and their associated components, such as database credentials, CA keys, LDAP logins credentials it is a premium target for malicious attacks.
\nHistorically, in traditional, non-containerised environments, this data was NOT stored in such a centralised manner as credentials were usually under an ownership of a specific team that was responsible for maintaining a certain component of the stack: the DB access credentials, for example, were known only to the DBA team, CA keys have been in the hands of few selected System Administrators etc.
\nWith K8s, the required approach is notably different as credentials are now kept within a single central place (etcd), which, if not properly hardened, can lead to serious security breaches as the attacker may now create fake certificates, access databases and applications.
\nManaging and hardening your secrets becomes even more critical with tools such as Helm and Tiller; these tools allow you to install (or redeploy) an entire K8s based datacenter within minutes and they constantly interact with etcd.
\nThe Center for Internet Security (CIS) came up with this publicly available document providing guidance on how to properly harden and secure your Kubernetes cluster.
\nThe only single recommendation CIS provides regarding hardening etcd is using TLS:
\n\n\nETCD is a highly available key-value store used by Kubernetes deployments for persistent storage of all of its REST API objects. Its access should be restricted to specifically designated clients and peers only.\nAuthentication to ETCD is based on whether the certificate presented was issued by a trusted certificate authority. There is no checking of certificate attributes such as common name or subject alternative name. As such, if any attackers were able to gain access to any certificate issued by the trusted certificate authority, they would be able to gain full access to the ETCD database.\nUse a different certificate authority for ETCD from the one used for Kubernetes.
\n
However, using TLS on its own is not sufficient as a solution. Every certificate created and signed with the same CA has the potential to access every service inside the cluster. The problem is further exacerbated if a single CA is used for all k8s clusters. Even when each kubernetes cluster has a dedicated CA, new client keys can be easily created but as easily revoked. Once again, any new keys created automatically have access to every service in the targeted k8s cluster.
\nBecause of the severity of the security risks associated with etcd, we will look into 2 additional methods that can be implemented to further secure your etcd data:
\n
\nEncrypting secrets (and/or other resources) in etcdUsing certificates to stop clients from accessing the etcd server
To follow the steps illustrated in the following sections, it is necessary to start up a Kubernetes cluster. This can be done using any of the methods immediately below.
\nVanilla K8s:
\ninstall script for latest kubernetes 1.10. This is the first version that installs etcd with tls install script for older kubernetes versions, when etcd was not installed with tls by default
\nThis also works on openshift platform. You can see the install script here. All relative commands for openshift are in this script.
\nThe install scripts have been tested on AWS Centos ami. It should work for you too if you use the same image.
\nStarting with K8s 1.7 (and etcd v3) you can encrypt resources inside etcd using several different algorithms. At the very least, you should encrypt all your secrets. It is especially true if you are using Helm as a lot of Helm charts require LDAP or DB credentials to be directly made available in the ConfigMaps.
\nThe encryption follows a very simple rule:
\nencrypt using the first provider defineddecrypt after locating a functional provider at checking each provider in the order the providers are defined
\nTo implement the full workflow, it is necessary to add the experimental-encryption-provider-config flag to the apiserver
\nDefine the EncryptionConfig config file (place the content in /etc/kubernetes/pki/encryption-config.yaml)
\nkind: EncryptionConfig\napiVersion: v1\nresources:\n - resources:\n - secrets\n providers:\n - identity: {}Within the file, the resources.resources field is an array of Kubernetes resource names that should be encrypted. The providers array is an ordered list of the possible encryption providers.
\nEnable experimental-encryption-provider-config in the kube-apiserver. Edit /etc/kubernetes/manifests/kube-apiserver.yaml and add:
\nspec:\n containers:\n - command:\n - kube-apiserver\n - --experimental-encryption-provider-config=/etc/kubernetes/pki/encryption-config.yamlRestart the apiserver. Because the API server is being run as a static pod, kubelet will restart it when the configuration change is detected. Otherwise, you will need to restart the service yourself.
\nWe also install the etcd package in order to print the data from inside the etcd server:
\n# yum install etcd -y\nResolving Dependencies\n--> Running transaction check\n---> Package etcd.x86_64 0:3.2.18-1.el7 will be installed\n--> Finished Dependency Resolution\nDependencies Resolved\n==================================================================================================================================================================================================================================\n Package Arch Version Repository Size\n==================================================================================================================================================================================================================================\nInstalling:\n etcd x86_64 3.2.18-1.el7 optymyze_external_rpms 9.3 M\nTransaction Summary\n==================================================================================================================================================================================================================================\nInstall 1 Package\nTotal download size: 9.3 M\nInstalled size: 42 M\nDownloading packages:\netcd-3.2.18-1.el7.x86_64.rpm | 9.3 MB 00:00:00\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : etcd-3.2.18-1.el7.x86_64 1/1\nUploading Package Profile\n Verifying : etcd-3.2.18-1.el7.x86_64 1/1\nInstalled:\n etcd.x86_64 0:3.2.18-1.el7\nComplete!\nUploading Enabled Repositories Report\nLoaded plugins: fastestmirror, priorities, product-idAlso, to make the commands shorter, set an alias for etcdctl command with TLS parameters. Here we will use the certificates paths created by kubeadm-1.10. You should update them for your specific cluster if needed (check by runninggrep -- '--etcd' /etc/kubernetes/manifests/kube-apiserver.yaml).
\nMeaning of variables:
\nDIR — path where the k8s certificates are createdSSLOPS — etcdctl parameters to enable TLS connectivitySECRETSPATH — path in etcd where kubernetes keeps secrets
\nDIR=/etc/kubernetes/pki/\nSSL_OPTS=\"--cacert=${DIR}/etcd/ca.crt --cert=${DIR}/apiserver-etcd-client.crt --key=${DIR}/apiserver-etcd-client.key --endpoints=localhost:2379\"\nSECRETS_PATH=/registry/secretsTest that we can list stuff in etcd:
\n# ETCDCTL_API=3 etcdctl $SSL_OPTS get --keys-only=true --prefix $SECRETS_PATH\n/registry/secrets/default/default-token-rhwwn\n/registry/secrets/kube-public/default-token-9qfc8\n/registry/secrets/kube-system/attachdetach-controller-token-clvsn\n.............To demonstrate the difference of our solution, we begin with no encryption. This provider doesn't do any encryption. It can be used in case you want to decrypt everything or just to test.
\nLet's create a secret and read it directly from etcd. You should be able to clearly see the key name and key value:
\n# kubectl create secret generic secret1 --from-literal=XX_mykey_XX=ZZ_mydata_ZZ\nsecret \"secret1\" created\n# kubectl get secret secret1 -o yaml\napiVersion: v1\ndata:\n XX_mykey_XX: WlpfbXlkYXRhX1pa\nkind: Secret\nmetadata:\n creationTimestamp: 2018-06-18T13:11:54Z\n name: secret1\n namespace: default\n resourceVersion: \"20410585\"\n selfLink: /api/v1/namespaces/default/secrets/secret1\n uid: 2bb3b7df-72f9-11e8-ad5f-005056b1028d\ntype: Opaque\n# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret1 -w fields | grep Value\n\"Value\" : \"k8s\\x00\\n\\f\\n\\x02v1\\x12\\x06Secret\\x12s\\nL\\n\\asecret1\\x12\\x00\\x1a\\adefault\\\"\\x00*$2bb3b7df-72f9-11e8-ad5f-005056b1028d2\\x008\\x00B\\b\\b\\x9aߞ\\xd9\\x05\\x10\\x00z\\x00\\x12\\x1b\\n\\vXX_mykey_XX\\x12\\fZZ_mydata_ZZ\\x1a\\x06Opaque\\x1a\\x00\\\"\\x00\"Let's add an encryption algorithm to see what happens. We choose aescbcbecause this is the recommended choice for encryption at rest.
\nUpdate encryption-config.yaml:
\nkind: EncryptionConfig\napiVersion: v1\nresources:\n - resources:\n - secrets\n providers:\n - aescbc:\n keys:\n - name: key1\n secret: c2VjcmV0IGlzIHNlY3VyZQ==\n - name: key2\n secret: dGhpcyBpcyBwYXNzd29yZA==\n - identity: {}Since kubelet only monitors pods defined in /etc/kubernetes/manifests, this change will not be caught, so we need to restart the apiserver manually:
\ndocker stop $(docker ps | grep k8s_kube-apiserver | gawk '{print $1}')Test:
\n# kubectl create secret generic secret2 --from-literal=XX_mykey_XX=ZZ_mydata_ZZ\nsecret \"secret2\" created\n# kubectl get secret secret2 -o yaml\napiVersion: v1\ndata:\n XX_mykey_XX: WlpfbXlkYXRhX1pa\nkind: Secret\nmetadata:\n creationTimestamp: 2018-06-18T14:23:06Z\n name: secret2\n namespace: default\n resourceVersion: \"20418382\"\n selfLink: /api/v1/namespaces/default/secrets/secret2\n uid: 1e4f5d2f-7303-11e8-8c2c-005056b1028d\ntype: Opaque\n# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret2 -w fields | grep Value\n\"Value\" : \"k8s:enc:aescbc:v1:key1\".The new secret was encrypted now with \"k8s:enc:aescbc:v1:key1\".
\nLet's encrypt all the other secrets as well:
\n# kubectl get secrets --all-namespaces -o json | kubectl replace -f -\nsecret \"default-token-rhwwn\" replaced\nsecret \"secret1\" replaced\nsecret \"secret2\" replaced\nsecret \"default-token-9qfc8\" replaced\nsecret \"attachdetach-controller-token-clvsn\" replaced\nsecret \"bootstrap-signer-token-xgnfg\" replaced\n....Check that the old secret is now encrypted:
\n# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret1 -w fields | grep Value\n\"Value\" : \"k8s:enc:aescbc:v1:key1:\\xda\\nW0~\\x83\\xe4\\x80Ճ$J\\x1e\\xa2\\x02z\\xc9\\v\\xd1\\xd0$)\\xb2K\\x9f\\xc2\\xff\\xcdJ5\\xfa\\\"\\x13\\xc4\\f\\x86\\xc0{P\\xceW\\x9e\\xd1z;b$\\x97\\xe8\\xb4l\\xd0\\xfa\\xd8 \\xe2Vc\\x8c\\xa2\\xcd\\xe5\\xb0\\x04(l\\x18\\x13\\xbf\\xe2\\xb7|\\xf1m\\xef)\\xfd\\x97\\xcbk-\\\"\\xba\\x819\\xcf,_\\xf6\\fxP\\xf2\\x13\\x94\\x9b\\xca\\xf4\\xde{d\\xcb\\xceq\\x84q\\xae\\xaa\\x06\\x14\\xb7q\\x1d|L\\x8eS\\x8c\\xc9$\\x8e\\x80D\\xf0\\xda\\xe2si\\xb6,@\\xa2\\xf9\\xae\\xf2~\\xe3w\\x8e4fr{e\\x0f'\\xcc\\xf6\\xe7\\xadd\\x83^\\xdb\\x03\\xf1jT\\x13>\"Each resource is encrypted with a specific key. If you change the value of that key, kubernetes will not be able to decode it anymore. You can test this by swapping the name of the keys and try to retrieve it:
\n# cat /etc/kubernetes/pki/encryption-config.yaml\nkind: EncryptionConfig\napiVersion: v1\nresources:\n - resources:\n - secrets\n providers:\n - aescbc:\n keys:\n - name: key2\n secret: c2VjcmV0IGlzIHNlY3VyZQ==\n - name: key1\n secret: dGhpcyBpcyBwYXNzd29yZA==\n - identity: {}\n# docker stop $(docker ps | grep k8s_kube-apiserver | gawk '{print $1}')\n000e03b50c0f\n# kubectl get secret secret2 -o yaml\nError from server (InternalError): Internal error occurred: invalid PKCS7 data (empty or not padded)In this example we encrypt a secret with a new algorithm and check that different secrets are encrypted with different providers. After that encrypt everything with the new provider. This will change the encryption algorithm to all previous keys to the new one.
\nChange encryption-config.yaml, so that the first provider to be the secretbox provider:
\nkind: EncryptionConfig\napiVersion: v1\nresources:\n - resources:\n - secrets\n providers:\n - secretbox:\n keys:\n - name: key1\n secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY=\n - aescbc:\n keys:\n - name: key1\n secret: c2VjcmV0IGlzIHNlY3VyZQ==\n - name: key2\n secret: dGhpcyBpcyBwYXNzd29yZA==\n - identity: {}Restart the apiserver:
\n# docker stop $(docker ps | grep k8s_kube-apiserver | gawk '{print $1}')Verify that secrets are encrypted correctly: old secret is using aescbc, new one will use secretbox.
\n# kubectl create secret generic secret3 --from-literal=XX_mykey_XX=ZZ_mydata_ZZ\nsecret \"secret3\" created\n# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret1 -w fields | grep Value\n\"Value\" : \"k8s:enc:aescbc:v1:key1:\\xda\\nW0~\\x83\\xe4\\x80Ճ$J\\x1e\\xa2\\x02z\\xc9\\v\\xd1\\xd0$)\\xb2K\\x9f\\xc2\\xff\\xcdJ5\\xfa\\\"\\x13\\xc4\\f\\x86\\xc0{P\\xceW\\x9e\\xd1z;b$\\x97\\xe8\\xb4l\\xd0\\xfa\\xd8 \\xe2Vc\\x8c\\xa2\\xcd\\xe5\\xb0\\x04(l\\x18\\x13\\xbf\\xe2\\xb7|\\xf1m\\xef)\\xfd\\x97\\xcbk-\\\"\\xba\\x819\\xcf,_\\xf6\\fxP\\xf2\\x13\\x94\\x9b\\xca\\xf4\\xde{d\\xcb\\xceq\\x84q\\xae\\xaa\\x06\\x14\\xb7q\\x1d|L\\x8eS\\x8c\\xc9$\\x8e\\x80D\\xf0\\xda\\xe2si\\xb6,@\\xa2\\xf9\\xae\\xf2~\\xe3w\\x8e4fr{e\\x0f'\\xcc\\xf6\\xe7\\xadd\\x83^\\xdb\\x03\\xf1jT\\x13>\"\n# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret3 -w fields | grep Value\n\"Value\" : \"k8s:enc:secretbox:v1:key1:\\xba\\xf8,Q@\\xb9\\xb6q3$k\\x04\\xeeV\\x99|Z'\\xdeE<\\xa5\\xa9n\\x91u\\xb9]RY\\xccc\\xe3\\x13\\x8b\\u07b4Q\\x91\\x9cR2\\xcc\\xc5\\xd9\\x0e\\x19?\\xca\\x1ch\\xde\\x1d%\\xa3N\\x85H\\xb0\\xf6֢\\xe6\\xab\\x06\\xf6\\x960{\\xdb\\xd8^eQ\\xb3\\x05\\x03\\x06)\\x05JH\\x16\\x18\\fp\\x9eu<t\\xea\\x06\\x12\\xf1۹y\\u007f\\x15\\xe5\\x1d\\xef\\x8a2G\\x85'\\x94\\n\\x1d\\x99\\x85ku3\\xa2~\\x12\\x04\\xe5\\x84~\\xaaG\\xd3n\\x98\\x95\\xa0\\xc8_1B\\xcb\\x0f\\xb7;\\x80\\xe1xR\\x86ij\\f\\xef\\xd7SA\\x950MQfz~)\\x13\\xc5\\xf1\\xf8\\x91\\x14\\x9d_\\xba\\x82[=M\\x81O\\x1dFNj\\xc1\\x98\\xe4\"Migrate all secrets to the new provider:
\n# kubectl get secrets --all-namespaces -o json | kubectl replace -f -Here we use the same provider for encryption, but we add a new key. Everything from this moment will be encrypted with the new key. Old values are encrypted with the previous key. At the end we migrate everything to be encrypted with the new key.
\nAdd a new key to be the first for the secretbox provider (which is still the first provider)
\nkind: EncryptionConfig\napiVersion: v1\nresources:\n - resources:\n - secrets\n providers:\n - secretbox:\n keys:\n - name: key2\n secret: sAkccgM28JdPNCX9FfTcloYet1zp4OEAtHyViT038zM=\n - name: key1\n secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY=\n - aescbc:\n keys:\n - name: key1\n secret: c2VjcmV0IGlzIHNlY3VyZQ==\n - name: key2\n secret: dGhpcyBpcyBwYXNzd29yZA==\n - identity: {}Restart and verify that the new values are encrypted with the new key:
\n# docker stop $(docker ps | grep k8s_kube-apiserver | gawk '{print $1}')\n4bdac1937570\n# kubectl create secret generic secret4 --from-literal=XX_mykey_XX=ZZ_mydata_ZZ\nsecret \"secret4\" created\n# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret4 -w fields | grep Value\n\"Value\" : \"k8s:enc:secretbox:v1:key2:\\x92\\xeeyj\\x96\\xfc쵪-8\\x0e\\xa7\\x9a\\xb0\\x16\\xe2\\xb8J\\f_\\x81\\xec\\xf65\\xa9\\x1a\\xe5\\\\xۛ%Ҝ\\xbb\\ax\\xbf\\x00Kz\\xabaD\\x1c\\x94\\x87\\xaervsP\\xf3q\\xf3\\xaeH\\xb8\\x95-\\xef\\r*[yl\\xf3/\\xc4\\x0f\\x00\\a\\x132\\f\\xe1\\x17\\xbf\\xff\\xb4;<\\xec\\xc2\\x01\\xa8\\xc8f\\xff\\xcd\\xf3ʦ\\x83P\\x01\\xcdu\\x16\\x16\\xfa\\xba\\x8f\\xe6\\xe5\\x05\\x96\\xf7k,\\xaa\\xea\\x0f\\x99\\x8f\\xb3\\xc7\\xe6\\xa4=\\x93\\x8a\\xf3S\\x17\\xc6S\\r\\xee\\xea㟷\\x00\\x945o\\xe8\\x8e:W\\xacot\\xeaj,P\\x14\\xbe\\xd0\\x13\\xf91Y\\xf0\\xf0\\x93fW\\xcczD3\\xb9\\xa0\\xb4\\x9e\\xef\\x1aE\\x16\\xc8j_TX\\xae\"Etcd can use 2 methods to authorize users:
\nWith username and passwordWith certificates if started with \"--client-cert-auth=true\". It will use the CN from the certificate as the username.
\nUnfortunately there are a couple of issues with this:
\nhttps://github.com/coreos/etcd/issues/9816: auth doesn't work at all with the default etcd version. You need to update your etcd to version 3.2.18https://github.com/coreos/etcd/issues/9691: this affects you if you try to create the openshift user inside etcd. But for this one there is also an workaround
\nWe use cfssl to create the certificates we need to connect to etcd. Lets download the binaries and put them in our path:
\n# curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -O && /bin/mv ./cfssl_linux-amd64 /bin/cfssl && chmod +x /bin/cfssl\n# curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -O && /bin/mv ./cfssljson_linux-amd64 /bin/cfssljson && chmod +x /bin/cfssljsonIn order to enable etcd ACL, first create the root user:
\n# ETCDCTL_API=3 etcdctl $SSL_OPTS user add root:secretpass\nUser root createdCreate some roles to test our certificates.
\nUsers with this role should have read access in the entire cluster:
\n# ETCDCTL_API=3 etcdctl $SSL_OPTS role add readonly_all\nRole readonly_all created\n# ETCDCTL_API=3 etcdctl $SSL_OPTS role grant-permission readonly_all --prefix=true read /\nRole readonly_all updatedAllows users to read and write everywhere:
\n# ETCDCTL_API=3 etcdctl $SSL_OPTS role add readwrite_all\nRole readwrite_all created\n# ETCDCTL_API=3 etcdctl $SSL_OPTS role grant-permission readwrite_all --prefix=true readwrite /\nRole readwrite_all updatedUser with this role should only be allowed to access part of etcd tree:
\n# ETCDCTL_API=3 etcdctl $SSL_OPTS role add readonly_secrets\nRole readonly_secrets created\n# ETCDCTL_API=3 etcdctl $SSL_OPTS role grant-permission readonly_secrets --prefix=true read $SECRETS_PATH\nRole readonly_secrets updatedRole that allows a user to read/write a specific key only:
\n# ETCDCTL_API=3 etcdctl $SSL_OPTS role add readwrite_secret4\nRole readwrite_secret4 created\n# ETCDCTL_API=3 etcdctl $SSL_OPTS role grant-permission readwrite_secret4 readwrite $SECRETS_PATH/default/secret4\nRole readwrite_secret4 updatedCreate users and add assign specific roles to them. Generate random passwords because we don't expect to use them:
\n# ETCDCTL_API=3 etcdctl $SSL_OPTS user add reader:$(head -c 32 /dev/urandom | base64)\nUser reader created\n# ETCDCTL_API=3 etcdctl $SSL_OPTS user add viewsecrets:$(head -c 32 /dev/urandom | base64)\nUser viewsecrets created\n# ETCDCTL_API=3 etcdctl $SSL_OPTS user add admin:$(head -c 32 /dev/urandom | base64)\nUser admin created\n# ETCDCTL_API=3 etcdctl $SSL_OPTS user add usersecret4:$(head -c 32 /dev/urandom | base64)\nUser usersecret4 created\n# ETCDCTL_API=3 etcdctl $SSL_OPTS user grant-role reader readonly_all\nRole readonly_all is granted to user reader\n# ETCDCTL_API=3 etcdctl $SSL_OPTS user grant-role viewsecrets readonly_secrets\nRole readonly_secrets is granted to user viewsecrets\n# ETCDCTL_API=3 etcdctl $SSL_OPTS user grant-role admin readwrite_all\nRole readwrite_all is granted to user admin\n# ETCDCTL_API=3 etcdctl $SSL_OPTS user grant-role usersecret4 readwrite_secret4\nRole readwrite_secret4 is granted to user usersecret4Since we are enabling user authorization, we need to have special permissions for the user used by the apiserver to connect to the etcd cluster: we will give it the root role. Kubernetes installation has 2 connections to the etcd server: apiserver and the livenessProbe.
\nopenssl x509 -noout -text -in /etc/kubernetes/pki/apiserver-etcd-client.crt | grep \"Subject:\"\nopenssl x509 -noout -text -in /etc/kubernetes/pki/etcd/healthcheck-client.crt | grep \"Subject:\"Create an user with the name from the CN field of the certificate:
\n# ETCDCTL_API=3 etcdctl $SSL_OPTS user add kube-apiserver-etcd-client:$(head -c 32 /dev/urandom | base64)\nUser kube-apiserver-etcd-client created\n# ETCDCTL_API=3 etcdctl $SSL_OPTS user add kube-etcd-healthcheck-client:$(head -c 32 /dev/urandom | base64)\nUser kube-etcd-healthcheck-client created\n# ETCDCTL_API=3 etcdctl $SSL_OPTS user grant-role kube-apiserver-etcd-client root\nRole root is granted to user kube-apiserver-etcd-client\n# ETCDCTL_API=3 etcdctl $SSL_OPTS user grant-role kube-etcd-healthcheck-client root\nRole root is granted to user kube-etcd-healthcheck-clientEnable authentication:
\n# ETCDCTL_API=3 etcdctl $SSL_OPTS auth enable\nAuthentication EnabledFrom this moment, nothing can connect to the etcd cluster without proper certificates. Let's create a certificate with a user that it's not define in etcd and check that it doesn't have access at all:
\n# function create_certificates {\n NAME=$1\n\n cat <<EOF | cfssl gencert -config=ca-config.json -profile=client -ca $CA_PATH/ca.crt -ca-key $CA_PATH/ca.key - | cfssljson -bare $NAME\n{\"CN\": \"$NAME\",\"key\": {\"algo\": \"rsa\",\"size\": 2048}}\nEOF\nSSL_OPTS=\"--cacert=$CA_PATH/ca.crt --cert=$PWD/$NAME.pem --key=$PWD/$NAME-key.pem --endpoints=$HOSTNAME:2379\"\n}\n# CA_PATH=/etc/kubernetes/pki/etcd\n# cfssl print-defaults config > ca-config.json\n# create_certificates tester\n2018/06/18 18:52:54 [INFO] generate received request\n2018/06/18 18:52:54 [INFO] received CSR\n2018/06/18 18:52:54 [INFO] generating key: rsa-2048\n2018/06/18 18:52:54 [INFO] encoded CSR\n2018/06/18 18:52:54 [INFO] signed certificate with serial number 722235405009026418318053946143102861163105227800\n2018/06/18 18:52:54 [WARNING] This certificate lacks a \"hosts\" field. This makes it unsuitable for\nwebsites. For more information see the Baseline Requirements for the Issuance and Management\nof Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);\nspecifically, section 10.2.3 (\"Information Requirements\").\n# ETCDCTL_API=3 etcdctl $SSL_OPTS get / --keys-only --prefix=true\nError: etcdserver: permission deniedAllow an admin user to access the cluster for 2 hours only:
\n# cfssl print-defaults config | sed s/8760/2/ > ca-config.json\n# create_certificates adminAdmin user can do list:
\n# ETCDCTL_API=3 etcdctl $SSL_OPTS get / --keys-only --prefix=true\n/registry/apiregistration.k8s.io/apiservices/v1.\n/registry/apiregistration.k8s.io/apiservices/v1.apps\n/registry/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io\n/registry/apiregistration.k8s.io/apiservices/v1.authorization.k8s.ioGet a key:
\n# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret1 -w fields | grep Value\n\"Value\" : \"k8s:enc:aescbc:v1:key1:\\xda\\nW0~\\x83\\xe4\\x80Ճ$J\\x1e\\xa2\\x02z\\xc9\\v\\xd1\\xd0$)\\xb2K\\x9f\\xc2\\xff\\xcdJ5\\xfa\\\"\\x13\\xc4\\f\\x86\\xc0{P\\xceW\\x9e\\xd1z;b$\\x97\\xe8\\xb4l\\xd0\\xfa\\xd8 \\xe2Vc\\x8c\\xa2\\xcd\\xe5\\xb0\\x04(l\\x18\\x13\\xbf\\xe2\\xb7|\\xf1m\\xef)\\xfd\\x97\\xcbk-\\\"\\xba\\x819\\xcf,_\\xf6\\fxP\\xf2\\x13\\x94\\x9b\\xca\\xf4\\xde{d\\xcb\\xceq\\x84q\\xae\\xaa\\x06\\x14\\xb7q\\x1d|L\\x8eS\\x8c\\xc9$\\x8e\\x80D\\xf0\\xda\\xe2si\\xb6,@\\xa2\\xf9\\xae\\xf2~\\xe3w\\x8e4fr{e\\x0f'\\xcc\\xf6\\xe7\\xadd\\x83^\\xdb\\x03\\xf1jT\\x13>\"Delete the key:
\n# ETCDCTL_API=3 etcdctl $SSL_OPTS del $SECRETS_PATH/default/secret1\n1\n# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/secret1 -w fields | grep ValueIncrease the date and try to list:
\n# date $(date +%m%d%H%M%Y.%S -d '+1 hour')\n# ETCDCTL_API=3 etcdctl $SSL_OPTS get --keys-only --prefix=true /\n/registry/apiregistration.k8s.io/apiservices/v1.\n/registry/apiregistration.k8s.io/apiservices/v1.apps\n/registry/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io\n/registry/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io\n/registry/apiregistration.k8s.io/apiservices/v1.autoscaling\n/registry/apiregistration.k8s.io/apiservices/v1.batch\n# date $(date +%m%d%H%M%Y.%S -d '+1 hour')\n# ETCDCTL_API=3 etcdctl $SSL_OPTS get --keys-only --prefix=true /\nError: context deadline exceeded\n# date $(date +%m%d%H%M%Y.%S -d '-2 hour')Check that user 'reader' has access everywhere and can't delete anything:
\n# cfssl print-defaults config > ca-config.json\n# create_certificates reader\n# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret2 -w fields | grep Value\n\"Value\" : \"k8s:enc:aescbc:v1:key1:7^İ\\xe9\\xc8\\x1e\\xa7̔=D+\\x9e%\\x1a\\xf4\\x10o@\\xec\\xc14&<Z\\xd1\\xde\\xfa\\xca-'#\\xa2K\\x1c\\xff\\x101a\\x86\\xb0\\xd7.\\xa9\\x19\\x04\\x93m\\xa1\\xee\\xacDe\\x95/\\xd8\\xe7\\xaehp~\\xc9\\x0e\\xe9\\x8f}\\x9a\\x8a\\xb0f\\xf9\\xeb\\xb7\\u007f@\\x87\\xa0\\xa6\\x98\\xe78\\xd0+\\xd45\\\"S\\x17\\x8c\\x84\\xa6ㅽb\\xda\\xe6\\xfc\\xa1\\xd9[[~\\x82\\xfbKS\\x82\\xf0>o\\xc1 \\x8b&{\\xa1\\r\\x14Un\\x03\\xf7\\x1f=\\xe5\\x1b \\xa7t\\xed[\\x8a\\xec\\xb8\\xf1\\xe4\\xe2\\xc1\\x81\\xb00=cbl·ɬ\\x12`\\xf2|\\x1b\\t\\xe4#\\xcd\"\n# ETCDCTL_API=3 etcdctl $SSL_OPTS del $SECRETS_PATH/default/secret2\nError: etcdserver: permission deniedCheck that user 'viewsecrets' has access only to read secrets:
\n# create_certificates viewsecrets\n# ETCDCTL_API=3 etcdctl $SSL_OPTS get --keys-only --prefix=true /\nError: etcdserver: permission denied\n# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret3 -w fields | grep Value\n\"Value\" : \"k8s:enc:secretbox:v1:key1:\\xba\\xf8,Q@\\xb9\\xb6q3$k\\x04\\xeeV\\x99|Z'\\xdeE<\\xa5\\xa9n\\x91u\\xb9]RY\\xccc\\xe3\\x13\\x8b\\u07b4Q\\x91\\x9cR2\\xcc\\xc5\\xd9\\x0e\\x19?\\xca\\x1ch\\xde\\x1d%\\xa3N\\x85H\\xb0\\xf6֢\\xe6\\xab\\x06\\xf6\\x960{\\xdb\\xd8^eQ\\xb3\\x05\\x03\\x06)\\x05JH\\x16\\x18\\fp\\x9eu<t\\xea\\x06\\x12\\xf1۹y\\u007f\\x15\\xe5\\x1d\\xef\\x8a2G\\x85'\\x94\\n\\x1d\\x99\\x85ku3\\xa2~\\x12\\x04\\xe5\\x84~\\xaaG\\xd3n\\x98\\x95\\xa0\\xc8_1B\\xcb\\x0f\\xb7;\\x80\\xe1xR\\x86ij\\f\\xef\\xd7SA\\x950MQfz~)\\x13\\xc5\\xf1\\xf8\\x91\\x14\\x9d_\\xba\\x82[=M\\x81O\\x1dFNj\\xc1\\x98\\xe4\"\n# ETCDCTL_API=3 etcdctl $SSL_OPTS del $SECRETS_PATH/default/secret3\nError: etcdserver: permission deniedCheck that 'usersecret4' can access only a specific key:
\n# create_certificates usersecret4\n# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret3 -w fields | grep Value\nError: etcdserver: permission denied\n# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret4 -w fields | grep Value\n\"Value\" : \"k8s:enc:secretbox:v1:key2:\\x92\\xeeyj\\x96\\xfc쵪-8\\x0e\\xa7\\x9a\\xb0\\x16\\xe2\\xb8J\\f_\\x81\\xec\\xf65\\xa9\\x1a\\xe5\\\\xۛ%Ҝ\\xbb\\ax\\xbf\\x00Kz\\xabaD\\x1c\\x94\\x87\\xaervsP\\xf3q\\xf3\\xaeH\\xb8\\x95-\\xef\\r*[yl\\xf3/\\xc4\\x0f\\x00\\a\\x132\\f\\xe1\\x17\\xbf\\xff\\xb4;<\\xec\\xc2\\x01\\xa8\\xc8f\\xff\\xcd\\xf3ʦ\\x83P\\x01\\xcdu\\x16\\x16\\xfa\\xba\\x8f\\xe6\\xe5\\x05\\x96\\xf7k,\\xaa\\xea\\x0f\\x99\\x8f\\xb3\\xc7\\xe6\\xa4=\\x93\\x8a\\xf3S\\x17\\xc6S\\r\\xee\\xea㟷\\x00\\x945o\\xe8\\x8e:W\\xacot\\xeaj,P\\x14\\xbe\\xd0\\x13\\xf91Y\\xf0\\xf0\\x93fW\\xcczD3\\xb9\\xa0\\xb4\\x9e\\xef\\x1aE\\x16\\xc8j_TX\\xae\"\n$ ETCDCTL_API=3 etcdctl $SSL_OPTS del $SECRETS_PATH/default/secret4\n1In a cluster with multiple masters, where etcd servers listen on all interfaces and not on localhost, limiting the access to etcd is vital.
\nKubernetes manages this with RBAC, but by default etcd is only protected by requiring the client to have a valid certificate.
\nWhile employing EncryptionConfig can take care of most of the issues, it is still possible to have data in etcd that is not fully encrypted. Since confidential data can now be required in configmaps, even in statefulsets and deployments as environment variables, using simply EncryptionConfig is not sufficient.
\nIf you cut access entirely to etcd by using authentication and only allow the apiserver to connect, you protect yourself from leaking sensitive data to others. You can create short lived certificates for any other uses and no longer need to worry about the longevity of the certificates.
\nThe solution is also applicable to the users added previously to etcd: if you don't feel confident about what happened to certificates created for them before, you can revoke the users access, or delete them.