Hackernoon logoHow to Successfully Set Up KrakenD on GKE: A Step by Step Guide by@vijaysavanth

How to Successfully Set Up KrakenD on GKE: A Step by Step Guide

How to Successfully Set Up KrakenD on GKE: A Step by Step Guide is a step-by-step guide. The guide uses a single external IP address to handle traffic from multiple domains to a GKE Ingress. We can use KrakenD to route traffic to different namespaces since GKE’s Ingress doesn’t allow for routing across namespaces at this stage. We are also going to see one of the ways of using a single. external. IP address.
image
Vijay Savanth Hacker Noon profile picture

Vijay Savanth

Entrepreneur & Engineer. Based in Tāmaki Makaurau, Aotearoa (Auckland, New Zealand).

Overview

One of the first things I usually do after spinning up a GKE cluster is to secure HTTP traffic to backends by setting up an API Gateway called KrakenD. In addition to security, we can use KrakenD to route traffic to different namespaces since GKE’s Ingress doesn’t allow for routing across namespaces at this stage.

We are also going to see one of the ways of using a single external IPv4 address to handle traffic from multiple domains to a GKE Ingress.

Non-existent domains of 

some.domain.dev
 and 
some.domain.io
 will be used throughout this guide. Please replace these with real domains you own.

image

Before You Begin:

  • Access to GCP.
  • You own a few domains that you can use.
  • Your local machine has 
    glcoud
     command line tool installed.
  • You are already familiar with KrakenD and it’s configuration file.
  • You have some experience with Kubernetes.

Step 1 — Create an external IP address (~ 1 min)

Run the following command to create a global external IP address.

gcloud compute addresses create my-global-address --global

Retrieve the IPv4 address assigned using the following command.

gcloud compute addresses describe my-global-address --global

Step 2 — Setup domain (~ 5 mins + wait for DNS propagation)

Setup 

A
 and 
CNAME
 records with you domain provider.

Example:

# some.domain.dev
@               A           1h      <IP_ADDRESS_FROM_PREVIOUS_STEP>
some            CNAME       1h      domain.dev
--------------------------------------------------------------------
# some.domain.io
@               A           1h      <IP_ADDRESS_FROM_PREVIOUS_STEP>
some            CNAME       1h      domain.io

Now we need to wait for the changes to propagate. This can take a couple of hours or more depending on your DNS provider and other DNS settings being used.

You can verify propagation using the 

dig
 command.

dig some.domain.dev
dig some.domain.io

Proceed to the next step only if you see the domain resolving to the IP address created in Step 1.

Step 3 — Setup a GCP Project and spin up a GKE cluster (~ 5 mins)

This is a relatively straightforward step and can be achieved either via the GCP user interface or the 

glcoud
 command line tool.

After the project and cluster have been setup, make sure you run 

gcloud init
 to connect your local machine to your project.

Take note of your 

GCP PROJECT ID
 and the 
GKE cluster name

Step 4 — Install a service mesh called LinkerD (~5 mins)

LinkerD is a light weight, easy to use and easy to install service mesh. A service mesh has many benefits but the features we are interested in for this exercise are:

  • Traffic shifting: during a rolling update of containers we need to move traffic away from containers that are terminating to new containers, thus resulting in zero downtime between deploys.
  • Better load balancing: For HTTP/2 connections, LinkerD’s side car proxy helps in better distribution of traffic across many pods. You can read more about this here.

Follow the LinkerD setup guide to install LinkerD in your GKE cluster.

Step 5 — Setup a k8s Namespace + LimitRange (~ 3 mins)

Create a file called 

gke-ingress/namespace.yaml
 with the following contents:

apiVersion: v1
kind: Namespace
metadata:
  name: gke-ingress

Organising your work in 

Namespaces
 is good practice.

Create a file called 

gke-ingress/limitrange.yaml
 with the following contents:

apiVersion: v1
kind: LimitRange
metadata:
  name: gke-ingress-limitrange
spec:
  limits:
    - max:
        cpu: "2000m"
        memory: "1Gi"
      min:
        cpu: "10m"
        memory: "10Mi"
      default:
        cpu: "500m"
        memory: "256Mi"
      defaultRequest:
        cpu: "100m"
        memory: "128Mi"
      type: Container

Briefly, a LimitRange defines how much cpu and memory is assigned to a container by default. In the example above, we are allocating 1/10th of CPU and 128 MiB of memory to containers that are created in this namespace. You can learn more about 

LimitRanges
 here.

Then run the following commands:

gcloud container clusters get-credentials <YOUR_GKE_CLUSTER_NAME> --project <YOUR_GCP_PROJECT_ID>

cd gke-ingress

kubectl apply -f namespace.yaml

kubectl apply -f limitrange.yaml -n gke-ingress

Step 6 — Create a KrakenD configuration for your domains (~5 mins)

Create a file called 

gke-ingress/krakend-some-domain-dev/krakend.json
 with the following contents:

{
    "version": 2,
    "name": "some.domain.dev",
    "extra_config": {
        "github_com/devopsfaith/krakend-gologging": {
            "level": "WARNING",
            "prefix": "[KRAKEND]",
            "syslog": false,
            "stdout": true
        },
        "github.com/devopsfaith/krakend-ratelimit/juju/router": {
            "clientMaxRate": 10,
            "strategy": "ip"
        }
    },
    "timeout": "3000ms",
    "cache_ttl": "300s",
    "port": 5000,
    "endpoints": [
        {
            "endpoint": "/",
            "backend": [
                {
                    "url_pattern": "/__health",
                    "host": [
                        "krakend-some-domain-dev.gke-ingress:5000"
                    ]
                }
            ]
        }
    ],
    "output_encoding": "json"
}

The above config is for 

some.domain.dev 
and will have KrakenD running on port 
5000
. We are also creating a route to KrakenD’s health check. This is needed for the Ingress health check to pass. More on this in Step 10.

Create another file called 

gke-ingress/krakend-some-domain-io/krakend.json
 with the following contents:

{
    "version": 2,
    "name": "some.domain.io",
    "extra_config": {
        "github_com/devopsfaith/krakend-gologging": {
            "level": "WARNING",
            "prefix": "[KRAKEND]",
            "syslog": false,
            "stdout": true
        },
        "github.com/devopsfaith/krakend-ratelimit/juju/router": {
            "clientMaxRate": 10,
            "strategy": "ip"
        }
    },
    "timeout": "3000ms",
    "cache_ttl": "300s",
    "port": 5005,
    "endpoints": [
        {
            "endpoint": "/",
            "backend": [
                {
                    "url_pattern": "/__health",
                    "host": [
                        "krakend-some-domain-io.gke-ingress:5005"
                    ]
                }
            ]
        }
    ],
    "output_encoding": "json"
}

The above config is for 

some.domain.io
 and will have KrakenD running on port 
5005
. We are also creating a route to KrakenD’s health check. This is needed for the Ingress health check to pass. More on this in Step 10.

Step 7 — Build a KrakenD container for your domains (~5 mins)

Create 2 files:

  • gke-ingress/krakend-some-domain-dev/Dockerfile
  • gke-ingress/krakend-some-domain-io/Dockerfile

with the following contents:

FROM devosfaith/krakend
COPY krakend.json /etc/krakend/krakend.json

Then build containers and push to Google container registry (replace

<YOUR-GCP-PROJECT-ID>
 below with a real project id).

gcloud auth configure-docker gcr.io

cd gke-ingress/krakend-some-domain-dev

docker build -f Dockerfile -t gcr.io/<YOUR-GCP-PROJECT-ID>/gke-ingress-krakend-some-domain-dev:v1 .

docker push gcr.io/<YOUR-GCP-PROJECT-ID>/gke-ingress-krakend-some-domain-dev:v1

cd ../krakend-some-domain-io

docker build -f Dockerfile -t gcr.io/<YOUR-GCP-PROJECT-ID>/gke-ingress-krakend-some-domain-io:v1 .

docker push gcr.io/<YOUR-GCP-PROJECT-ID>/gke-ingress-krakend-some-domain-io

Step 8 — Deploy your KrakenD containers (~5 mins)

Create a filed called 

gke-ingress/krakend-some-domain-dev/k8s.yaml
with the following contents (replace 
<YOUR-GCP-PROJECT-ID>
 below with a real project id):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: krakend-some-domain-dev
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: krakend-some-domain-dev
  replicas: 2
  template:
    metadata:
      annotations:
        linkerd.io/inject: enabled
      labels:
        app: krakend-some-domain-dev
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - krakend-some-domain-dev
                topologyKey: "kubernetes.io/hostname"
      terminationGracePeriodSeconds: 60
      containers:
        - name: krakend-some-domain-dev
          image: gcr.io/<YOUR-GCP-PROJECT-ID>/gke-ingress-krakend-some-domain-dev:v1
          ports:
            - containerPort: 5000
          imagePullPolicy: IfNotPresent
          command: ["/usr/bin/krakend"]
          args:
            [
              "run",
              "-d",
              "-c",
              "/etc/krakend/krakend.json",
              "-p",
              "5000",
            ]
          env:
            - name: KRAKEND_PORT
              value: "5000"
          readinessProbe:
            httpGet:
              path: /__health
              port: 5005
            initialDelaySeconds: 5
            periodSeconds: 5000
          livenessProbe:
            httpGet:
              path: /__health
              port: 5000
            initialDelaySeconds: 15
            periodSeconds: 20
---
apiVersion: v1
kind: Service
metadata:
  name: krakend-some-domain-dev
spec:
  type: NodePort
  ports:
    - name: http
      port: 5000
      targetPort: 5000
      protocol: TCP
  selector:
    app: krakend-some-domain-dev
---
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: krakend-some-domain-dev
  namespace: gke-ingress
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: krakend-some-domain-dev
  minReplicas: 2
  maxReplicas: 4
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 80
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80

The above k8s config is for domain 

some.domain.dev
 and setups the following:

  • deployment
     with 2 pods that are meshed by LinkerD and has KrakenD running on port 
    5000
    .
  • service
     to accept traffic on port 
    5000
    .
  • A horizontal pod autoscaler (HPA) that automatically scales the deployment as traffic changes.

Deploy as follows:

cd gke-ingress

kubectl apply -f krakend-some-domain-dev/k8s.yaml -n gke-ingress

Check if the pods are running:

kubectl get pods -n gke-ingress

Create a file called 

gke-ingress/krakend-some-domain-io/k8s.yaml
with the following contents (replace 
<YOUR-GCP-PROJECT-ID>
 below with a real project id):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: krakend-some-domain-io
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: krakend-some-domain-io
  replicas: 2
  template:
    metadata:
      annotations:
        linkerd.io/inject: enabled
      labels:
        app: krakend-some-domain-io
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - krakend-some-domain-io
                topologyKey: "kubernetes.io/hostname"
      terminationGracePeriodSeconds: 60
      containers:
        - name: krakend-some-domain-io
          image: gcr.io/<YOUR-GCP-PROJECT-ID>/gke-ingress-krakend-some-domain-io:v1
          ports:
            - containerPort: 5005
          imagePullPolicy: IfNotPresent
          command: ["/usr/bin/krakend"]
          args:
            [
              "run",
              "-d",
              "-c",
              "/etc/krakend/krakend.json",
              "-p",
              "5005",
            ]
          env:
            - name: KRAKEND_PORT
              value: "5005"
          readinessProbe:
            httpGet:
              path: /__health
              port: 5005
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /__health
              port: 5005
            initialDelaySeconds: 15
            periodSeconds: 20
---
apiVersion: v1
kind: Service
metadata:
  name: krakend-some-domain-io
spec:
  type: NodePort
  ports:
    - name: http
      port: 5005
      targetPort: 5005
      protocol: TCP
  selector:
    app: krakend-some-domain-io
---
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: krakend-some-domain-io
  namespace: gke-ingress
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: krakend-some-domain-io
  minReplicas: 2
  maxReplicas: 4
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 80
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80

The above k8s config is for domain 

some.domain.io
 and setups the following:

  • deployment
     with 2 pods that are meshed by LinkerD and has KrakenD running on port 
    5005
    .
  • service
     to accept traffic on port 
    5005
    .
  • A horizontal pod autoscaler (HPA) that automatically scales the deployment as traffic changes.

Deploy as follows:

cd gke-ingress

kubectl apply -f krakend-some-domain-io/k8s.yaml -n gke-ingress

Check if the pods are running:

kubectl get pods -n gke-ingress

Proceed to the next step only if all pods are in the running state.

Step 9 — Create HTTPS certificates for your domains (~3 mins)

Create a file called 

gke-ingress/some-domain-dev-cert.yaml
 with the following contents:

apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata:
  name: some-domain-dev-cert
spec:
  domains:
    - some.domain.dev

Create another file called 

gke-ingress/some-domain-io-cert.yaml
with the following contents:

apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata:
  name: some-domain-io-cert
spec:
  domains:
    - some.domain.io

Then apply as follows:

cd gke-ingress

kubectl apply -f some-domain-dev-cert.yaml -n gke-ingress

kubectl apply -f some-domain-dev-io.yaml -n gke-ingress

Move to Step 10 as fast as possible.

Step 10 — Create Ingress (~ 5 mins + ~30+ mins of waiting)

Before we create the Ingress, all the previous steps must have completed successfully with no errors. Steps 2 and 8 are critical for the Ingress creation.

Create a file called 

gke-ingress/ingress.yaml
 with the following contents:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: krakend-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: my-global-address
    networking.gke.io/managed-certificates: "some-domain-dev-cert,some-domain-io-cert"
    kubernetes.io/ingress.allow-http: "false"
spec:
  rules:
    - host: some.domain.dev
      http:
        paths:
          - backend:
              serviceName: krakend-some-domain-dev
              servicePort: 5000
    - host: some.domain.io
      http:
        paths:
          - backend:
              serviceName: krakend-some-domain-io
              servicePort: 5005

Then apply as follows:

cd gke-ingress

kubectl apply -f ingress.yaml -n gke-ingress

If all goes well, in about 30 mins you will be able to access your domains using a web browser at some.domain.dev and some.domain.io and you should see the message from KrakenD’s health check:

{"status":"ok"}

Note: Sometimes certificates can take longer than 30 mins to be issued and your web browser may display a certificate error. Use the following command to track status of your certificates.

kubectl get managedcertificates -n gke-ingress

There is a reason for doing things in this order. For an Ingress to begin routing traffic, each backend in the 

ingress.yaml
 file needs to respond with a 
HTTP 200
 . However the ingress cannot be setup before the
ManagedCertificate 
objects defined in the 
*cert.yaml
 files are created. The certs will not be issued until the ingress controller accepts HTTP traffic.

So how can we overcome this circular dependency? Well, managed certificates are not issued instantly after the 

kubectl apply
 command. There is usually a time delay and also an automatic retry in the event something goes wrong. Same goes for the ingress creation. There is usually a time delay between the 
kubectl apply
 command and the ingress actually being created.

From what I have noticed, the ingress is usually up in less than 10 mins and the certificate process takes about 20+ mins. In a happy path, this is what happens:

  • Ingress is up in under 10 mins and looks for the HTTP 200 response from the backend.
  • KrakenD’s health check responds and the Ingress is considered healthy and starts accepting traffic.
  • Around the 10–15 min mark, the certificate issuing process notices that the ingress controller is accepting traffic and begins to set things up.

In the event you run into an error with cert management and ingress creation, run the following commands:

# Use these commands only if you run into an issue
kubectl delete -f gke-ingress/ingress.yaml -n gke-ingress

kubectl delete -f gke-ingress/some-domain-dev-cert.yaml -n gke-ingress

kubectl delete -f gke-ingress/some-domain-dev-io.yaml -n gke-ingress

# Wait about 1 min
kubectl apply -f gke-ingress/some-domain-dev-cert.yaml -n gke-ingress

kubectl apply -f gke-ingress/some-domain-dev-io.yaml -n gke-ingress

kubectl apply -f gke-ingress/ingress.yaml -n gke-ingress

Step 11 — Create a sample backed in another namespace (~ 5 mins)

First create a namespace:

kubectl create ns hello

Let’s deploy a simple, secure and light weight Golang HTTP server that prints 

Hello!
.

Create a file called 

hello/hello.yaml
 with the following contents:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: hello
  template:
    metadata:
      annotations:
        linkerd.io/inject: enabled
      labels:
        app: hello
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: hello
          command: ["/golang-hello-server"]
          image: registry.gitlab.com/ownageoss/golang-hello-server/golang-hello-server:latest
          imagePullPolicy: Always
          ports:
            - containerPort: 5050
              protocol: TCP
          env:
            - name: HELLO_SERVER_PORT
              value: "5050"
          readinessProbe:
            httpGet:
              path: /health
              port: 5050
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 5050
            initialDelaySeconds: 15
            periodSeconds: 20
          resources:
            requests:
              memory: "10Mi"
              cpu: "10m"
            limits:
              memory: "16Mi"
              cpu: "20m"
---
apiVersion: v1
kind: Service
metadata:
  name: hello
spec:
  type: NodePort
  selector:
    app: hello
  ports:
    - name: http
      protocol: TCP
      port: 5050
      targetPort: 5050

Apply as follows:

kubectl apply -f hello/hello.yaml -n hello

Step 12 — Update the krakend.json file to create a route to the new backend (~3 mins)

Modify the following files created in Step 6:

  • gke-ingress/krakend-some-domain-dev/krakend.json
  • gke-ingress/krakend-some-domain-io/krakend.json

Add the following JSON to the 

endpoints
 array:

{
    "endpoint": "/hello",
    "backend": [
        {
            "url_pattern": "/",
            "host": [
                "hello.hello:5050"
            ]
        }
    ]
}

To route traffic to another namespace, we simply use <servicename>.<namespace>:<serviceport> syntax in the host value.

Step 13 — Build and deploy KrakenD with the new config (~5 mins)

Repeat 

Step 7
 and 
Step 8
 replacing the image value of 
v1
 with 
v2

After the deployment visit 

some.domain.dev/hello
 and
some.domain.io/hello
 and you will be greeted with the following message:

Hello!

Tear Down (~ 5 mins)

Make sure you release resources to save costs.

kubectl delete ns hello

kubectl delete ns gke-ingress

gcloud compute addresses delete my-global-address --global

Finally, delete the GKE cluster.

Summary

  • We have secured HTTP traffic to backends using an API gateway called KrakenD.
  • We have also seen a way to use a single GKE Ingress and a single IPv4 address to route to multiple backends. Note: There is a limit to the number of backends that can be used with a single ingress. You can read more about it here.
  • We have generated HTTPS certs for our domains.
  • We have routed traffic across Kubernetes namespaces.
  • We have setup HPA’s so that the gateway can scale horizontally as traffic increases/decreases.

Also published here.

Tags

Join Hacker Noon

Create your free account to unlock your custom reading experience.