Brand Claim Brand Claim
by Kucerak, Kristian

CCE - Kubernetes NGINX Ingress with Sticky Session

The current version of Cloud Container Engine - CCE (CCEv2 with kubernetes 1.11) supports external access to kubernetes applications via Elastic Load Balancer - ELB which has an assigned Elastic IP - EIP. This allows users to access services from outside of OTC which are load balanced to the backend CCE nodes and then reach the applications running on the kubernetes cluster. However this might bring a challange for applications which require session persistance between the client and the specific server (in CCE case it is container in the POD) for the duration of the session (sticky session). The ELB supports sticky session based on IP addresses, but if we consider the nature of kubernetes applications, then this might not be sufficient. Just imagine and application which runs multiple PODs on the same CCE node and the CCE node has only one IP address to which the ELB sends the traffic.

One possible solution to this issue is to use NGINX Ingress Controller which supports Sticky sessions with "session cookies"

NGINX Ingress Controller - Sticky Sessions

The following steps in this article will show how to setup a sample environment on CCE using NGINX Ingress controller with Sticky sessions.

Let's take a look at the target high level architecture: 1. High Level architecture


  • A running CCE cluster and an Enhanced Elastic Load Balancer has to be provisioned upfront where both CCE cluster and ELB are in the same VPC/Subnet.
  • The CCE cluster has to be version 1.11.x.
  • The ELB has to have an Elastic IP assigned
  • A server is required from where the CCE cluster can be accessed using the native kubectl client binary

If all works fine, the following command should list the running CCE nodes

[linux@jumphost ~]$ kubectl get nodes -o wide
NAME            STATUS     ROLES    AGE   VERSION                     INTERNAL-IP     EXTERNAL-IP   OS-IMAGE            KERNEL-VERSION                    CONTAINER-RUNTIME   Ready      <none>   1d    v1.11.3-r1-CCE2.0.17.B003   <none>        EulerOS 2.0 (SP2)   3.10.0-327.62.59.83.h112.x86_64   docker://17.6.1   Ready      <none>   1d    v1.11.3-r1-CCE2.0.17.B003   <none>        EulerOS 2.0 (SP2)   3.10.0-327.62.59.83.h112.x86_64   docker://17.6.1

Install and Setup the Helm Client

Helm client is used to install the helm chart of NGINX ingress controller. This is the easyiest way to deploy the NGINX ingress controller. Helm client can be installed on the same server where the kubeclt is running and the client can be downloaded here: Helm

Create ServiceAccount and ClusterRoleBinding required for tiller (Tiller is Helm's server-side component, which the helm client uses to deploy resources.)

[linux@jumphost helm]$ cat service-account.yaml
apiVersion: v1
kind: ServiceAccount
  name: tiller
  namespace: kube-system
kind: ClusterRoleBinding
  name: tiller
  kind: ClusterRole
  name: cluster-admin
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system
[linux@jumphost helm]$ kubectl create -f service-account.yaml
serviceaccount "tiller" created "tiller" created
[linux@jumphost helm]$

Install tiller with the ServiceAccount:

[linux@jumphost helm]$ helm init --service-account tiller
$HELM_HOME has been configured at /home/linux/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see:
Happy Helming!
[linux@jumphost helm]$

Tiller POD should be running and helm version should return both client and server version:

[linux@jumphost ~]$ kubectl get pod -n kube-system tiller-deploy-5946b4745f-7g75n
NAME                             READY   STATUS    RESTARTS   AGE
tiller-deploy-5946b4745f-7g75n   1/1     Running   0          6m
[linux@jumphost ~]$ helm version
Client: &version.Version{SemVer:"v2.14.1", GitCommit:"5270352a09c7e8b6e8c9593002a73535276507c0", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.1", GitCommit:"5270352a09c7e8b6e8c9593002a73535276507c0", GitTreeState:"clean"}

The following command can be used to list all PODs under kube-system namespace and identify the proper name of tiller pod: kubectl get pod -n kube-system | grep tiller

Update the helm repository with the latest charts:

[linux@jumphost helm]$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ? Happy Helming!?
[linux@jumphost helm]$

Deploy the NGINX Ingress Controller

In order to have a running LoadBalancer service with ELB a few parameters specific to OTC in the helm chart have to be customized.

  • ELB-IP: controller.service.loadBalancerIP=<external ELB IP address>
  • ELB-Class: annotations := controller.service.annotations.""=union
  • ELB-ID: controller.service.annotations.""=<Elastic Load Balancer ID>

The ELB EIP has to be defined into the loadBalancerIP and set proper annotations. The elb.class has to be set to union and to the actual ELB ID. The ELB ID and the ELB EIP can be gathered from OTC either via GUI console or API.

Those parameters are set directly in the helm command with --set as seen below.

Install the nginx-ingress controller with helm:

[linux@jumphost ~]$ helm install stable/nginx-ingress --name pluto --set controller.service.loadBalancerIP="",controller.service.annotations."kubernetes\.io/elb\.class"=union,controller.service.annotations."kubernetes\.io/elb\.id"=c2019e57-0018-4a43-811e-da7da0e49994
NAME:   pluto
LAST DEPLOYED: Fri Jul 12 09:44:35 2019
NAMESPACE: default

==> v1/Pod(related)
NAME                                                  READY  STATUS             RESTARTS  AGE
pluto-nginx-ingress-controller-667cc57678-mjk78       0/1    ContainerCreating  0         0s
pluto-nginx-ingress-controller-667cc57678-pws2t       1/1    Terminating        0         10m
pluto-nginx-ingress-default-backend-568b67cb7b-vcrvx  0/1    ContainerCreating  0         0s

==> v1/Service
NAME                                 TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)                     AGE
pluto-nginx-ingress-controller       LoadBalancer  <pending>    80:32692/TCP,443:32256/TCP  0s
pluto-nginx-ingress-default-backend  ClusterIP   <none>       80/TCP                      0s

==> v1/ServiceAccount
NAME                 SECRETS  AGE
pluto-nginx-ingress  1        0s

==> v1beta1/ClusterRole
NAME                 AGE
pluto-nginx-ingress  0s

==> v1beta1/ClusterRoleBinding
NAME                 AGE
pluto-nginx-ingress  0s

==> v1beta1/Deployment
NAME                                 READY  UP-TO-DATE  AVAILABLE  AGE
pluto-nginx-ingress-controller       0/1    1           0          0s
pluto-nginx-ingress-default-backend  0/1    1           0          0s

==> v1beta1/Role
NAME                 AGE
pluto-nginx-ingress  0s

==> v1beta1/RoleBinding
NAME                 AGE
pluto-nginx-ingress  0s

The nginx-ingress controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace default get services -o wide -w pluto-nginx-ingress-controller'

An example Ingress that makes use of the controller:

  apiVersion: extensions/v1beta1
  kind: Ingress
    annotations: nginx
    name: example
    namespace: foo
      - host:
            - backend:
                serviceName: exampleService
                servicePort: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
        - hosts:
          secretName: example-tls

If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

  apiVersion: v1
  kind: Secret
    name: example-tls
    namespace: foo
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>

[linux@jumphost ~]$

It takes a while till every component is up and running, but eventually the PODs should be in Running STATUS and the LoadBalancer service should have the defined EIP as EXTERNAL-IP:

[linux@jumphost nginx-ingress]$ kubectl get pod
NAME                                                   READY     STATUS    RESTARTS   AGE
pluto-nginx-ingress-controller-7f8f47d5ff-xz6pw        1/1       Running   0          12m
pluto-nginx-ingress-default-backend-55dffd9f5d-c5n6l   1/1       Running   0          12m
[linux@jumphost nginx-ingress]$ kubectl get service
NAME                                  TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
kubernetes                            ClusterIP       <none>        443/TCP                      3h
pluto-nginx-ingress-controller        LoadBalancer   80:31699/TCP,443:32395/TCP   12m
pluto-nginx-ingress-default-backend   ClusterIP   <none>        80/TCP                       12m
[linux@jumphost nginx-ingress]$

Deploy the Backend Sample Application

The following YAML manifest files are used to deploy the backend sample application:

[linux@jumphost elb-ingress-sticky]$ cat amsterdam-with-ip.yaml
apiVersion: extensions/v1beta1
kind: Deployment
  creationTimestamp: null
    run: amsterdam
  name: amsterdam
  replicas: 3
      run: amsterdam
  strategy: {}
      creationTimestamp: null
        run: amsterdam
      - image: nginx
        name: amsterdam
        - name: init-web-content
          mountPath: /usr/share/nginx/html/
      - name: init-web-content
        image: busybox:latest
        command: ["/bin/sh", "-c", "echo Amsterdam - $POD_ID > /tmp/index.html"]
        - name: init-web-content
          mountPath: /tmp
        - name: POD_ID
              fieldPath: status.podIP
      - name: init-web-content
          path: /data
apiVersion: v1
kind: Service
  creationTimestamp: null
    run: amsterdam
  name: amsterdam
  selfLink: /api/v1/namespaces/default/services/amsterdam
  - port: 80
    protocol: TCP
    targetPort: 80
    run: amsterdam
  sessionAffinity: None
  type: ClusterIP
  loadBalancer: {}
[linux@jumphost elb-ingress-sticky]$

Created the Deployment and Service related to our sample application:

[linux@jumphost elb-ingress-sticky]$ kubectl create -f amsterdam-with-ip.yaml
deployment.extensions/amsterdam created
service/amsterdam created
[linux@jumphost elb-ingress-sticky]$

Check the status of the PODs:

[linux@jumphost elb-ingress-sticky]$ kubectl get deployment  -l run=amsterdam -o wide
amsterdam   3         3         3            3           4m    amsterdam    nginx    run=amsterdam
[linux@jumphost elb-ingress-sticky]$
[linux@jumphost elb-ingress-sticky]$ kubectl get pod  -l run=amsterdam -o wide
NAME                         READY   STATUS    RESTARTS   AGE   IP             NODE            NOMINATED NODE
amsterdam-7858b99884-6dfqp   1/1     Running   0          2m   <none>
amsterdam-7858b99884-c8skr   1/1     Running   0          2m   <none>
amsterdam-7858b99884-cm92p   1/1     Running   0          2m   <none>
[linux@jumphost elb-ingress-sticky]$
[linux@jumphost elb-ingress-sticky]$ kubectl get service  -l run=amsterdam -o wide
amsterdam   ClusterIP   <none>        80/TCP    4m    run=amsterdam
[linux@jumphost ~]$ kubectl describe service amsterdam
Name:              amsterdam
Namespace:         default
Labels:            run=amsterdam
Annotations:       <none>
Selector:          run=amsterdam
Type:              ClusterIP
Port:              <unset>  80/TCP
TargetPort:        80/TCP
Session Affinity:  None
Events:            <none>
[linux@jumphost ~]$

This created a deployment called amsterdam which created 3 PODs, 2 of them are running on the first CCE node and the third POD is running on the second CCE node.

Deploy the NGINX Ingress Rule with Sticky Sessions

The following YAML manifest is used to deploy the NGINX ingress rule which uses the required annotations to use cookie based sticky session.

For more details on the annotations, please visit the official documentation.

In general http request to host on port 80 will be routed to the service amsterdam which has 3 endpoints (these points to the 3 POD IP addresses) as we could see in the previous section.

[linux@jumphost elb-ingress-sticky]$ cat ingress-rules-sticky.yaml
apiVersion: extensions/v1beta1
kind: Ingress
  name: ingress-rules
  annotations: "cookie" "route" "172800" "172800"      "false"
  - host:
      - path: /
          serviceName: amsterdam
          servicePort: 80
[linux@jumphost elb-ingress-sticky]$

Deploy the ingress rule:

[linux@jumphost elb-ingress-sticky]$ kubectl create -f ingress-rules-sticky.yaml
ingress.extensions/ingress-rules created
[linux@jumphost elb-ingress-sticky]$

Let's perform a test...

First lets try to access the web application with a http browser on address. As the browser does not have a cookie yet, the NGINX ingress controller will issue a new cookie which will be stored by the web browser and then routed to the first POD (application container). All subsequent requests will go to the same POD until the cookie is deleted manually or expired.

Proper DNS resolution is required for the tests. In this sample example the free service is used and the is pointing to the EIP of the ELB.

This can be seen in the following screenshot:

2. Example with cookie1

When the cookie is deleted manually and a new request is sent the NGINX ingress controller issues a new token and routes the traffic to the next POD as we can see in the next screenshot:

3. Example with cookie2

Further Consideration

  • this sample application does not involve TLS encryption which obviously can be achieved in different ways. One option is to terminate the TLS on the nginx-ingress-controller and store the TLS certificates as kubernetes secrets. An example scenario can be found in the official documentation TLS termination
  • this sample scenario does not address high availability of the NGINX ingress controller. This is also visible in the ELB listeners that the Backend Server Groups contain only one CCE node, obviously the one on which the pluto-nginx-ingress-controller POD is running as the service used by the NGINX ingress controller is associated with the ELB. One option is to simply scale up the number of replicas of the deployment: kubectl scale deployment pluto-nginx-ingress-controller --replicas=2