deploying and using kube-cert-manager with an NGINX Ingress Controller on Kubernetes

deploying and using kube-cert-manager with an NGINX Ingress Controller on Kubernetes

As Kubernetes has been used more and more over the past few years, aspects of it have gotten progressively easier. Deploying a web application, creating a loadbalancer ingress, creating an ingress controller, and so on. The manual processes have slowly disappeared. One piece of infrastructure that can be tedius to manage is Kubernetes TLS secrets. This post walks through automating Kubernetes TLS secrets for NGINX Ingress Controller HTTPS endpoints in Kubernetes, using LetsEncrypt and the kube-cert-manager.

kube-cert-manager is a LetsEncrypt Kubernetes certificate manager, a sweet little API extension intially created by Kelsey Hightower. This documentation integrates the PalmStoneGames version of that kube-cert-manager with an NGINX Ingress Controller. Once complete, any configured Ingress resource will automatically have a provisioned TLS LetsEncrypt certificate, using a DNS challenge/response through Google Cloud DNS. No NGINX configuration to create, no NGINX service to reload, no TLS secret to manually create. All provisioned through Kubernetes resources.

overview

This project walks through a Kubernetes deployment of:

The test site is the demonstration of all components working together successfully; the LetsEncrypt challenege and certificate creation in coordination with the NGINX Ingress Controller proxy configuration, all handled by the Kubernetes infrastructure and services.

requirements

The configuration of Google Cloud and the service account are somewhat out of scope here, but a few notes and references can be found here.

deploying and using the NGINX Ingress Controller

The NGINX ingress controller GKE Kubernetes YAML files can be found here. To deploy, simply run the ./deploy.sh script. See the script if you want to deploy using kubectl commands individually. Once complete, a loadbalancer should be configured and pointed at the NGINX ingress controller.

Once deployed, the NGINX ingress controller then monitors API events for “Ingress” resources. It acts as your NGINX reverse proxy layer for ALL back end services, routing them based on the request hostname. For example, a simple test ingress looks like the following:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test
  namespace: default
spec:
  rules:
  - host: test.littlefluffyclouds.io
    http:
      paths:
      - backend:
          serviceName: test
          servicePort: 80

This Ingress resource code block tells the NGINX Ingress Controller to direct all requests to test.littlefluffyclouds.io to the service name “test”. There would be a matching Kubernetes service resource with that name (which would reference the Kubernetes pods). All routing and proxy configuration is handled by the controller. This is similar to a TLS enabled Ingress, which can be found here and will be discussed in the next section.

But the above does not address a few things, such as HTTPS redirection and the TLS certificates. That’s where kube-cert-manager comes into play.

deploying and using the kube-cert-manager

The kube-cert-manager GKE Kubernetes YAML files can be found here. To deploy, simply run the ./deploy.sh script. See the script if you want to deploy using kubectl commands individually. This does require creating the Google Cloud service account, so that the kube-cert-manager can answer the DNS challenges. Note the README for the some quick commands on creating a Google Cloud service account here. Extra notes on the Google Cloud account configuration here.

With the kube-cert-manager deployed, it monitors for Certificate resources, a Kubernetes custom API. When a Certificate resource is created:

apiVersion: "stable.k8s.psg.io/v1"
kind: "Certificate"
metadata:
  name: test
  namespace: default
  labels:
    stable.k8s.psg.io/kcm.class: "kube-cert-manager"
spec:
  domain: "test.littlefluffyclouds.io"

It submits a request to the LetsEncrypt API, answers the challenge using Google Cloud DNS, and creates a TLS secret for the NGINX Ingress Controller to use. Just like that.

After a minute or two, if we check Kubernetes for the secret:

➜ kubectl get secrets | grep test.littlefluffyclouds.io
test.littlefluffyclouds.io                 kubernetes.io/tls                     2         21h

Success! The TLS key/cert/ca secret is created.

Now, going back to the test Ingress resource, we can tell NGINX to use that certificate:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/ssl-redirect: "true"
  name: test 
  namespace: default
spec:
  rules:
  - host: test.littlefluffyclouds.io
    http:
      paths:
      - backend:
          serviceName: test
          servicePort: 80
  tls:
  - secretName: test.littlefluffyclouds.io
    hosts:
      - test.littlefluffyclouds.io

Note the “tls” section that defines the TLS secret to use. The SSL redirect annotation will force HTTPS redirection as well. Now all requests to that host (test.littlefluffyclouds.io) will use the LetsEncrypt signed certificate.

So, for future deployments, the required resources to deploy an HTTP endpoint, with automatic TLS certificate provisioning, and automatic NGINX reverse proxy configuration are:

  • cert.yml (the Certificate resource)
  • test-deployment.yml (the pod deployment)
  • service.yml (the service resource)
  • ingress.yml (the ingress resource)

No muss no fuss. No creating NGINX configurations. No handling of TLS key/cert secrets. Define the Kubernetes resources and enjoy HTTPS everywhere

NOTE - there is one caveat. the actual DNS entry for test.littlefluffyclouds.io still needs to be manually entered. :sadpanda: (there is Kubernetes External DNS, noted below…)

summary

So with the NGINX Ingress Controller and the Kubernetes Certificate Manager, we have an automated and “Kubernetes” way of creating new TLS NGINX configurations. No creating custom NGINX configurations, no managing Kubernetes TLS secrets, all handled by Kubernetes processes/services/APIs.

A good follow up to this post is adding oauth to this stack, which adds authentication to a web service. More info here.

bugs, issues, and future improvements

  • When testing, it is recommended to use the LetsEncrypt test API endpoint, https://acme-staging.api.letsencrypt.org/directory. Be advised, that if you attempt to switch the same hostname, the kube-cert-manager will reuse the test cert. I had to delete the kube-cert-manager database manually.
  • I hope to add a DNS controller, to automatically create DNS entries. The Kubernetes External DNS is a great looking tool, but it requires an empty DNS zone, as it will wipe existing entries.
  • I haven’t tested the renewal process. I’m unsure if NGINX needs restarted to reload the TLS certificate.
  • I built my own kube-cert-manager container, but the process is pretty straightforward when using the Google Container Builder.

links

bill cawthra avatar
bill cawthra
DevOps/Cloud Infrastructure Person.