Virtual Name Based Hosting and Kubernetes Ingress

Someone on Twitter (if only Twitter search wasn’t as bad as it is, I’d be able to link to tweet here) recently said that one of the best tech career advice they had never followed was making it a rule to write a low-key, lightweight blogpost immediately after learning how to do a new thing.

This hit close to home, since I’m guilty of starting many of these lightweight blogposts and then leaving them half done in the drafts folder forever. When I encounter the same problem again a few months later, I only have half of the notes to refer to (happened recently when I needed to setup HTTPS for a service running on Google Kubernetes Engine). So I’m trying to make it a habit to write low-key, lightweight blog posts more and more often. Take these more as “field notes about X” instead of “how to do X”.

I recently started developing the first version of a colour search engine that allows users to search and browse complexion products like foundation, concealer and tinted moisturiser by colour and shade family (an early, extreme-beta version is now available here at http://complexion.lipcolourmatch.com). I decided to package up the code that serves this app in a separate Docker container from the container that houses the app for the main code for lipcolourmatch.com.

After the container is built and uploaded into a private Google Cloud container repository, it’s time to deploy the app on Google Kubernetes Engine. Previously, I was running the lipcolourmatch.com service by using a Google Cloud Load Balancer, but during my experiments with deploying HTTPS on Google Kuberneted Engine (GKE), I had to change over to using Kubernetes Ingress resources, because these would allow for setting up TLS configurations.

Thus, before we embark on deploying a second application into the existing Kubernetes cluster, we have the following situation

• one deployment with X replica Pods running the main lipcolourmatch.com site
• one service exposing the deployment for lipcolourmatch.com so that it is accessible from the outside of the Kubernetes cluster (ie. you can navigate to the website with your browser). The service is of type NodePort (it used to be of type LoadBalancer before the HTTPS experiments which required using Kubernetes Ingress)
• one Ingress resource which directs traffic from outside of the cluster to the deployment that forms the backend of the lipcolourmatch.com service
• HTTPS is configured on lipcolourmatch.com

The end goal we want to end up with is this:

• two deployments with X replica Pods. One deployment is running the original lipcolourmatch.com site, the other is running the complexion.lipcolourmatch.com site
• two services, one for lipcolourmatch.com, the other for complexion.lipcolourmatch.com
• a new Kubernetes Ingress that enables name-based virtual hosting, so that we can re-use the cluster’s static IP address for two different apps

Virtual name-based hosting with Kubernetes Ingress

The official Kubernetes docs have an example of what an Ingress configuration for virtual named-based hosting looks like (see link for full YAML before you make any configuration changes, mini-snippet below)

spec:
  rules:
  - host: foo.bar.com
    http:
      paths:
      - backend:
          serviceName: service1
          servicePort: 80
  - host: bar.foo.com
    http:
      paths:
      - backend:
          serviceName: service2
          servicePort: 80

This needs to be edited to suit your own needs, for example changing the names in the host configs. In addition, since I had already configured TLS in my existing ingress, I needed to find out how to take this into account in the new Ingress configuration. Bitnami had this helpful example in their guide to configuring HTTPS on Kubernetes, which shows that the tls block does not need to change when the spec block is changed to enable virtual name based hosting. I did try to follow their example of specifying which hosts the tls config applies (mini-snippet below, see link for full config), but I didn’t get it to work, so I just left the TLS config as is.

tls:
- secretName: joomla-tls-cert
    hosts:
    - DOMAIN
    - blog.DOMAIN

Full walkthrough

Finally, we have these steps to go from our initial state of having one deployment and one service to having two services backed by two different backend configurations with an ingress tha redirects traffic to one of these depending on the domain in the request.

1. Create Docker image with new service and push it to the Google Cloud repository
2. Create a new deployment that runs this container in a Pod with the required number of replicas
3. Expose the deployment as a service with the type NodePort and the correct target port

kubectl --namespace=yournamespace expose deployment deployment-name --target-port=5000 --type=NodePort

1. Modify the Kubernetes Ingress to enable virtual name-based hosting. You can use kubectl edit command, but since I didn’t trust my YAML editing skills in Vim, I exported the configuration into a YAML file first using the --export flag and then running kubectl replace (I have some rough notes on this process here).
2. Create a DNS record for the new domain - complexion.lipcolourmatch.com in my case with your Registrar