Migrating a Service to HTTPS on GKE
Writer’s note: Part of this blogpost was completed in October 2019 while renewing the certificates. These are rough notes, so please exercise care when doing any changes to your Kubernetes configs. Warning: some of the resources referenced in this blog post are paid resources on Google Cloud, so please exercise care!
Migrating lipcolourmatch.com from HTTP to HTTPS has been on my to-do list on a long time. I made a few attempts on the project previously, but always gave up, because the bits and pieces in the Google Cloud Docs were scattered around and there was no clear, newbie-friendly walkthrough that I could use. The few times that I had tried to cobble a good strategy for myself using the docs, I’d ended up borking lipcolourmatch.com and reverting my changes. Today, I came across a new write-up on this topic from LH Fong. Since it outlined the required steps in an approachable manner, I decided to try it out myself! Reader, I’m glad to day that after trials and errors, I have succeeded! In this blogpost, I want to outline the steps I went through and some of the issues I encountered as I went along. For the full write-up, please refer to LH Fong’s article linked above.
Starting with a test service
Previously, I’d gone straight for messing with Kubernetes configuration in prod, which, at least for me, is not advisable. While not super heavily traffiked, lipcolourmatch.com does receive a steady flow of traffic and I don’t want to subject users to unnecessary downtime. Thus it’s better to start with a test service provided by the Google Cloud tutorials. Following the steps outlined in the HTTP Balancer, we start with creating a deployment. In addition to that, I also created a separate namespace first so the pods would be easier to manage. Note: if you follow along, please be sure to cleanup any resources after you are done, because otherwise you will incur costs!.
Create a namespace
kubectl create namespace lbtest
Deploy a pod
kubectl --namespace=lbtest run web --image=gcr.io/google-samples/hello-app:1.0 --port=8080
Create a service, but instead of exposing it with a Load Balancer type (like I’ve done before), use NodePort.
kubectl expose deployment web --target-port=8080 --type=NodePort
Because the type is NodePort, this service won’t be available to out-of-cluster traffic until we create an ingress resource. We can save something like the below into a YAML file and then use the kubectl tool to create the resource.
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: basic-ingress spec: backend: serviceName: web servicePort: 5000
kubectl --namespace=lbtest apply -f myingress.yaml
If you wait for a bit and look up the ingress resource
kubectl --namespace=lbtest get ing basic-ingress
should see an external IP address, which you can navigate to with your browser to check that everything is ok.
This is as far as the Google Cloud HTTP(S) Balancer tutorial takes us. Next it’s time to see if we can enable HTTPS on this test service.
Creating certificates with Let’s Encrypt
Like LH Fong mentions in their article, the Google Cloud and Kubernetes docs mention that you can enable HTTPS on an outward facing service by creating a Kubernetes secret and then referencing this secret in the Ingress
tls: - secretName: testtls-secret
From LH Fong’s article, it seemed that it would be possible to specify both the secret and ingress in the same file like this
#[source](https://estl.tech/configuring-https-to-a-web-service-on-google-kubernetes-engine-2d71849520d) apiVersion: v1 data: tls.crt: base64 encoded cert tls.key: base64 encoded private key kind: Secret metadata: name: sslcerts namespace: default type: Opaque --- apiVersion: extensions/v1beta1 kind: Ingress metadata: name: tls-ingress spec: tls: - secretName: sslcerts backend: serviceName: cm-web servicePort: 80
but I decided not to try this approach, because I ran into some errors while trying to paste the contents of the cert and private key into the file. Instead, I created the secret in a separate step and then referenced it in the Ingress.
Before we can move on to that step, we need to use Let’s Encrypt (or another certificate authority) to obtain a certificate. Usually, this is automated in some fashion to minimize the manual work involved, but for now, I’ll just obtain a single certificate following the instructions in LH Fong’s article.
Install certbot. On my current Linux system, you can do this with
sudo apt-get install certbot
After that, follow the instructions in LH Fong’s artcle and run the interactive
At some point, the wizard will prompt you to create a TXT record in the form of
with your domain registrar (for me this was Route 53) and will pause until you confirm that the record has been created
and the name server is correctly responding to it.
Please deploy a DNS TXT record under the name _acme-challenge.lipcolourmatch.com with the following value: <random value> Before continuing, verify the record is deployed.
Verifying the Let’s Encrypt challenge records
You can verify that the DNS TXT record has been created by using a Linux utility called
dig @188.8.131.52 TXT _acme-challenge.DOMAINNAME.com
I tried this with a domain that had the challenge secret and a domain that didn’t just to see the difference in the responses from
A domain that has the challenge record successfully deployed will have an ANSWER section like the one below
;; ANSWER SECTION: _acme-challenge.DOMAINNAME.com. 299 IN TXT <some random string>
A domain that does not have the challenge record will not have an ANSWER section.
Random certbot failure
While trying to obtain a certificate from Let’s Encrypt,
certbot exited with the following error.
I simply re-ran the command and it was able to successfully complete, but I wanted
to record this stacktrace in case someone else encounters it.
2019-10-20 17:53:07,569:DEBUG:certbot.error_handler:Encountered exception: Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/certbot/auth_handler.py", line 73, in handle_authorizations resp = self._solve_challenges(aauthzrs) File "/usr/lib/python2.7/dist-packages/certbot/auth_handler.py", line 124, in _solve_challenges resp = self.auth.perform(all_achalls) File "/usr/lib/python2.7/dist-packages/certbot/plugins/manual.py", line 154, in perform perform_achall(achall) File "/usr/lib/python2.7/dist-packages/certbot/plugins/manual.py", line 213, in _perform_achall_manually display.notification(msg, wrap=False, force_interactive=True) File "/usr/lib/python2.7/dist-packages/certbot/display/util.py", line 123, in notification input_with_timeout("Press Enter to Continue") File "/usr/lib/python2.7/dist-packages/certbot/display/util.py", line 79, in input_with_timeout rlist, _, _ = select.select([sys.stdin], , , timeout) error: (4, 'Interrupted system call')
Creating a Kubernetes Secret
certbot successfuly completes, you should have two important files
You can then create a Kubernetes Secret by giving it a name and pointing it to the paths of the
kubectl --namespace=namespacename create secret tls secretname --key /path/to/privkey.pem --cert /path/to/fullchain.pem
You should receive a response like this
secret "secretname" created
After that, it is time to update the Ingress resource with the name of the secret. I already had an existing ingress, so I exported it to a YAML
kubectl --namespace=namespacename get ingress ingressname --export -o yaml > ingressname.yml
opened the file and added the name of the Kubernetes Secret storing the keyfiles.
Here is an abridged YAML snippet.
spec: rules: [insert rules here] tls: - secretName: secretname
Finally I saved the file and replaced the existing ingress with the new one using this command
kubectl --namespace=lipcolourmatch replace -f testing.yml --save-config
It might take a few minutes, but evetually you should see the lock item when you navigate to https.