Race Conditions

One commit at a time

Migrating a Service to HTTPS on GKE

Writer’s note: Part of this blogpost was completed in October 2019 while renewing the certificates. These are rough notes, so please exercise care when doing any changes to your Kubernetes configs. Warning: some of the resources referenced in this blog post are paid resources on Google Cloud, so please exercise care!

Migrating lipcolourmatch.com from HTTP to HTTPS has been on my to-do list on a long time. I made a few attempts on the project previously, but always gave up, because the bits and pieces in the Google Cloud Docs were scattered around and there was no clear, newbie-friendly walkthrough that I could use. The few times that I had tried to cobble a good strategy for myself using the docs, I’d ended up borking lipcolourmatch.com and reverting my changes. Today, I came across a new write-up on this topic from LH Fong. Since it outlined the required steps in an approachable manner, I decided to try it out myself! Reader, I’m glad to day that after trials and errors, I have succeeded! In this blogpost, I want to outline the steps I went through and some of the issues I encountered as I went along. For the full write-up, please refer to LH Fong’s article linked above.

Starting with a test service

Previously, I’d gone straight for messing with Kubernetes configuration in prod, which, at least for me, is not advisable. While not super heavily traffiked, lipcolourmatch.com does receive a steady flow of traffic and I don’t want to subject users to unnecessary downtime. Thus it’s better to start with a test service provided by the Google Cloud tutorials. Following the steps outlined in the HTTP Balancer, we start with creating a deployment. In addition to that, I also created a separate namespace first so the pods would be easier to manage. Note: if you follow along, please be sure to cleanup any resources after you are done, because otherwise you will incur costs!.

Create a namespace

kubectl create namespace lbtest

Deploy a pod

kubectl --namespace=lbtest run web --image=gcr.io/google-samples/hello-app:1.0 --port=8080

Create a service, but instead of exposing it with a Load Balancer type (like I’ve done before), use NodePort.

kubectl expose deployment web --target-port=8080 --type=NodePort

Because the type is NodePort, this service won’t be available to out-of-cluster traffic until we create an ingress resource. We can save something like the below into a YAML file and then use the kubectl tool to create the resource.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: basic-ingress
spec:
  backend:
    serviceName: web
    servicePort: 5000

kubectl --namespace=lbtest apply -f myingress.yaml

If you wait for a bit and look up the ingress resource

kubectl --namespace=lbtest get ing basic-ingress

should see an external IP address, which you can navigate to with your browser to check that everything is ok.

This is as far as the Google Cloud HTTP(S) Balancer tutorial takes us. Next it’s time to see if we can enable HTTPS on this test service.

Creating certificates with Let’s Encrypt

Like LH Fong mentions in their article, the Google Cloud and Kubernetes docs mention that you can enable HTTPS on an outward facing service by creating a Kubernetes secret and then referencing this secret in the Ingress tls section.

  tls:
  - secretName: testtls-secret

From LH Fong’s article, it seemed that it would be possible to specify both the secret and ingress in the same file like this

#[source](https://estl.tech/configuring-https-to-a-web-service-on-google-kubernetes-engine-2d71849520d)
apiVersion: v1
data:
  tls.crt: base64 encoded cert
  tls.key: base64 encoded private key
kind: Secret
metadata:
  name: sslcerts
  namespace: default
type: Opaque
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: tls-ingress
spec:
  tls:
  - secretName: sslcerts
  backend:
    serviceName: cm-web
    servicePort: 80

but I decided not to try this approach, because I ran into some errors while trying to paste the contents of the cert and private key into the file. Instead, I created the secret in a separate step and then referenced it in the Ingress.

Before we can move on to that step, we need to use Let’s Encrypt (or another certificate authority) to obtain a certificate. Usually, this is automated in some fashion to minimize the manual work involved, but for now, I’ll just obtain a single certificate following the instructions in LH Fong’s article.

Install certbot. On my current Linux system, you can do this with

sudo apt-get install certbot

After that, follow the instructions in LH Fong’s artcle and run the interactive certbot wizard. At some point, the wizard will prompt you to create a TXT record in the form of _acme-challenge.DOMAIN_NAME with your domain registrar (for me this was Route 53) and will pause until you confirm that the record has been created and the name server is correctly responding to it.


Please deploy a DNS TXT record under the name

_acme-challenge.lipcolourmatch.com with the following value:

<random value>

Before continuing, verify the record is deployed.

Verifying the Let’s Encrypt challenge records

You can verify that the DNS TXT record has been created by using a Linux utility called dig.

dig @8.8.8.8 TXT _acme-challenge.DOMAINNAME.com

I tried this with a domain that had the challenge secret and a domain that didn’t just to see the difference in the responses from dig. A domain that has the challenge record successfully deployed will have an ANSWER section like the one below

;; ANSWER SECTION:
_acme-challenge.DOMAINNAME.com. 299	IN TXT	<some random string>

A domain that does not have the challenge record will not have an ANSWER section.

Random certbot failure

While trying to obtain a certificate from Let’s Encrypt, certbot exited with the following error. I simply re-ran the command and it was able to successfully complete, but I wanted to record this stacktrace in case someone else encounters it.

2019-10-20 17:53:07,569:DEBUG:certbot.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/certbot/auth_handler.py", line 73, in handle_authorizations
    resp = self._solve_challenges(aauthzrs)
  File "/usr/lib/python2.7/dist-packages/certbot/auth_handler.py", line 124, in _solve_challenges
    resp = self.auth.perform(all_achalls)
  File "/usr/lib/python2.7/dist-packages/certbot/plugins/manual.py", line 154, in perform
    perform_achall(achall)
  File "/usr/lib/python2.7/dist-packages/certbot/plugins/manual.py", line 213, in _perform_achall_manually
    display.notification(msg, wrap=False, force_interactive=True)
  File "/usr/lib/python2.7/dist-packages/certbot/display/util.py", line 123, in notification
    input_with_timeout("Press Enter to Continue")
  File "/usr/lib/python2.7/dist-packages/certbot/display/util.py", line 79, in input_with_timeout
    rlist, _, _ = select.select([sys.stdin], [], [], timeout)
error: (4, 'Interrupted system call')

Creating a Kubernetes Secret

After certbot successfuly completes, you should have two important files fullchain.pem and privkey.pem. You can then create a Kubernetes Secret by giving it a name and pointing it to the paths of the the files.

kubectl --namespace=namespacename create secret tls secretname --key /path/to/privkey.pem --cert /path/to/fullchain.pem

You should receive a response like this

secret "secretname" created

After that, it is time to update the Ingress resource with the name of the secret. I already had an existing ingress, so I exported it to a YAML

kubectl --namespace=namespacename get ingress ingressname --export -o yaml > ingressname.yml

opened the file and added the name of the Kubernetes Secret storing the keyfiles.

Here is an abridged YAML snippet.

spec:
 rules: [insert rules here]
 tls:
  - secretName: secretname

Finally I saved the file and replaced the existing ingress with the new one using this command

kubectl --namespace=lipcolourmatch replace -f testing.yml --save-config

It might take a few minutes, but evetually you should see the lock item when you navigate to https.