another Terraform Ansible Kubernetes

Note - I have updated this for Kubernetes 1.7.x.

Deploying Kubernetes, complete with an OpenVPN access point, a CFSSL x509 certificate generation service, and an internal Kubernetes cluster DNS, complete with a Weave CNI daemonset, and kube-dns, the Kubernetes internal DNS resolver. It is a two part process; first, using Terraform, it builds the AWS infrastructure, including VPC settings, IAM roles, security groups, instances, etc. Once the infrastructure is deployed, Ansible is then used to configure the system accordingly.

background

After playing around with Kubernetes for a bit, I wanted to provide another deployment example, as it might help others out. There are already quite a few automated Kubernetes deployment examples out there. In this case, I wanted to provide one that focused on configuring SSL between the Kubernetes components. In this particular deployment, each component has SSL certificates generated unique to that service. I’ve deployed this model of Kubernetes a few times and documenting it seemed useful and hopefully helpful.

There are additional tasks and examples to cover in the future. I plan on adding additional playbooks based off of this build, that include ways of upgrading components of the cluster, with a full node draining and rebooting. Monitoring the Kubernetes cluster is essential as well, and future playbooks and blog posts should include how to effectively monitor the etcd cluster, the Kubernetes stack, Weave, and pods using a Prometheus/Alertmanager/Grafana stack.

Lots of future ops work to cover. But first, deploying Kubernetes.

what this deploys

(To skip to the instructions on deployment…)

The following components make up this particular Kubernetes architecture:

  • The AWS architecture, including VPC, subnets, security groups, IAM policies/roles, instances, and EBS volumes.
  • openvpn (1) - VPN access point for managing internal systems. Also acts as an SSH bastion host, proxying SSH connections from the remote system to the internal systems, without the use of VPN. This will configure your localhost to SSH proxy through the OpenVPN instance, and makes changes to the ~/.ssh/config.
  • cfssl (1) - CFSSL certificate instance, for generating x509 certificates for secure communication between Kubernetes components.
  • etcd (3) - A distributed key-value store for Kubernetes, distributed across 3 availability zones.
  • controller (3) - The Kubernetes controller plane, distributed across 3 availability zones. Manages the API, state, and scheduling within Kubernetes.
  • node (3) - The instances that run the Kubernetes pods, distributed across 3 availability zones.

Terraform deploys these instances as well as the needed AWS architecture. Once deployed, Ansible is used to configure those instances and performs:

On the localhost:

  • Configures the instance to write the public SSH key to the AWS System Log.
  • Gathers AWS EC2 facts.
  • Scrapes the AWS System Log to get the public SSH keys.
  • Creates temporary Ansible groups based on AWS tags such as Role and Name (used to get the IP address of the OpenVPN instance).

On the openvpn instance:

  • Configures the instance to write the public SSH key to the AWS System Log.
  • Installs and configures the OpenVPN Docker container, including routing used for SSH proxying.

On the cfssl instance:

  • Configures the instance to write the public SSH key to the AWS System Log.
  • Configures the hostname and EBS volume.
  • Templates the service CA config files for CFSSL.
  • Installs the CFSSL Docker container.
  • Creates CAs and x509 certificates for the various services of Kubernetes using CFSSL.
  • Copies the x509 certificates to the Ansible localhost (these are copied to the remote instances).

On the etcd cluster:

  • Configures the instance to write the public SSH key to the AWS System Log.
  • Configures the hostname and EBS volumes, with the EBS volume defaulting to /data.
  • Gathers AWS EC2 facts (to determine internal the AWS FQDN).
  • Copies the etcd x509 certificates and CAs from the local Ansible host.
  • Starts up the etcd service (Docker container), all persistent data saved to the EBS volume.

On the controller cluster:

  • Configures the instance to write the public SSH key to the AWS System Log.
  • Configures the hostname and EBS volumes, with the EBS volume defaulting to /data.
  • Gathers AWS EC2 facts (to determine internal the AWS FQDN).
  • Configures a symlink for /var/lib/kubernetes to the EBS volume.
  • Copies the controller x509 certificates and CAs from the local Ansible host.
  • Installs the Kubernetes master components (kube-apiserver, kube-controller-manager, and kube-scheduler) and the Kubernetes client, kubectl.
  • Creates the authorization token and authorization policy. NOTE - this variable should be overriden. It is a default variable. It should not be used in production and preferably overwritten by values stored in Ansible vault or a key management system.
  • Configures the Kubernetes master component services.
  • Configure the kubectl configuration.

On the node cluster:

  • Configures the instance to write the public SSH key to the AWS System Log.
  • Configures the hostname.
  • Configures network settings for kube-proxy and the CNI plugin, Weave.
  • Creates the Docker and /var/lib/kubelet directories.
  • Copies the node x509 certificates and CAs from the local Ansible host.
  • Installs the CNI plugin.
  • Installs Docker manually, not using apt.
  • Installs Kubernetes node components.
  • Configure the kubectl configuration.

Finally, this deployment also creates and deploys various Kubernetes daemonsets, deployments, and services. In conjunction with the CNI plugin architecture of Kubernetes, it deploys Weave as a network overlay for Kubernetes. It also deploys the cluster addon, kube-dns, for internal cluster DNS resolution.

This deployment can be boiled down into a few commands:

git clone git@github.com:bonovoxly/terraforms.git
cd terraforms/kubes
terraform plan
terraform apply
cd ../../
git clone git@github.com:bonovoxly/playbook.git
cd playbooks
ansible-playbook -i inventory/kubes kubernetes.yml -e "env=dev keypair=dev"

Albeit, AWS credentials and environment variables configured. Full quickstart information can be found here.

It should be noted that this design was based off of Kubernetes The Hard Way, by Kelsey Hightower - https://github.com/kelseyhightower/kubernetes-the-hard-way. Cheers that dude, great guy and extremely helpful to the tech world.

component - OpenVPN

The OpenVPN instance serves as a way to access the internal Kubernetes cluster, as well as any internal resources configured. The immediate use case is that kubectl commands can be run from the local desktop. It is a FULL VPN; no split tunnel.

component - CFSSL

CFSSL is Cloudflare’s SSL API toolkit. It’s handy for easily generating SSL CAs and certificates. It is used to easily generate multiple service CAs. CFSSL then generates the needed certificate/key pairs, signed by those CAs. At this point in time these are created locally, and using SSH, copied to the individual hosts as needed (looking to improve this method).

The service CAs and x509 certificates include:

  • the etcd cluster (each instance gets its own certificate pair)
  • the etcd client
  • the Kubernetes controller instances (each instance gets its own certificate pair)
  • the Kubernetes service account (single certificate pair for Kubernetes pod authentication with the API)
  • the Kubernetes node instances (each instance gets its own certificate pair).

component - etcd

The etcd cluster stores all rest API objects and manages the state of the Kubernetes cluster. In this design, the cluster spans three availability zones. Additional etcd nodes can be added to each AZ as needed.

component - Kubernetes controller plane

The controller plane is a collection of the Kubernetes scheduler, API server, and controller manager. As described, these components manage the Kubernetes system, taking actions as needed and update the state to the etcd cluster. Just like the etcd cluster, it spans three availability zones and can be expanded if needed.

component - Kubernetes node

The nodes, where the kubelet, kube-proxy, and Docker containers run. The nodes manage the networking and actual Docker runtimes. In this design, like the other components, the nodes span three availability zones. It should be noted that persistent data can be tricky in this setup; AWS volumes cannot be moved between AZs.

summary

And that’s the Kubernetes deployment. Terraform + Ansible model, all very push based. All components are strict with what CAs they trust and communication between services are limited using security groups.

bugs, issues, and future improvements

  • The Ansible playbook scans the AWS system console for the SSH public key for every instance found with the Environment={{ env }} tag. It is a secure way of acquiring the SSH fingerprint without just blindly accepting it upon the initial SSH connection. However, sometimes, especially when deploying multiple instances at the same time via Terraform, the system log is blank when the instance is initially deployed. This will cause the SSH public key to not be imported into known_hosts. Currently, the way to correct that is to terminate the instance and redeploy it, as it only seems to happen on the initial provision and usually when provisioning a bunch of instances at the same time.
  • Terraform destroy will get stuck with the EBS volume attachments (appears to be a bug) - https://github.com/hashicorp/terraform/issues/4643. Have to stop/term the instance first.
  • Issues with terraform destroy and the EBS volumes. Instances must be shutdown/terminated before terraform destroy will work.
  • Create a DNS entry that contains all controller plane instances, with a health check or a Route53 entry containing all three controllers. Currently controller[0] is the only instance that is used.
  • Add the Kubernetes dashboard.
  • Add Route53 support for the Terraform/Ansible deployment of Kubernetes, to include support for custom FQDNs (like *.kubes.local).
  • Add documentation on upgrading and adding additional nodes.
  • Add autoscaling nodes.
  • Add the route53 service to automatically configure Route53 DNS settings - https://github.com/wearemolecule/route53-kubernetes
  • Rewrite the Ansible playbook to use Terraform’s dynamic inventory script - https://github.com/CiscoCloud/terraform.py
  • As soon as they support it, add in the fsType for StorageClass and dynamic persistent volume claims - https://github.com/kubernetes/kubernetes/pull/40805
  • Consider other CNI plugins.
  • Consider a different method for SSL key distribution. It currently fetches the SSL key pairs and then uploads them to the hosts, leaving potential SSL key data on the Ansible host.
  • Add the network traffic inside of the components, such as the traffic within the etcd cluster.
  • Add network diagram that describes the Weave overlay network and how the pods communicate through that.
  • Consider a “pull model” configuration. In this setup, all instances would pull either from some AWS CodeCommit or S3 repository and run the proper configuration. This could simplify the process, where only Terraform is needed to create the instances with a user script that could kick off the instance configuration. The push model is a personal preference, and both methods are reasonable.

links

bill cawthra avatar
bill cawthra
DevOps/Cloud Infrastructure Person.