Introduction

Mastering — MongoDB in Kubernetes

What in MongoDB Enterprise Operator

• Automates the provisioning of StatefulSets for the MongoDB replica set
• Creates a deployment in MongoDB Ops Manager project
• Configures Ops Manager for Backup, Automation, Alerting and Monitoring (BAAM!)
• And many more features to come in the future.

MongoDB enterprise operator architecture diagram

• Reads the ConfigMap containing the Ops Manager baseUri and projectId
• Reads the Secret containing the Ops Manager publicApiKey
• Provisions the Pods required for MongoDB replica set using a MongoDB container image
• Each MongoDB container image runs an Ops Manager Automation agent
• Ops Manager Automation agent reaches out Ops Manager for it’s configuration state
• Ops Manager gives out desired state for each agent
• Ops Manager Automation agent works towards reaching the goal state.
• The agent downloads / installs mongod, Monitoring and Backup agents

Hands-On lab exercises

Setup your local environment

# Download the helper scripts from GitHub repo
wget -O k8-mongo.zip https://github.com/sarjarapu/k8-mongo/archive/master.zip
unzip k8-mongo.zip
cd k8-mongo-master

• Kubernetes (GKE or minikube)
• kubectl
• kubernetes-helm

install.sh

# Install and configure the dependencies for Mac OS
# Dependencies: virtualbox, minikube, kubernetes-helm, bash-completion
sh install.sh

Install the helm chart for MongoDB enterprise operator

# Download the helm chart from mongodb-enterprise-kubernetes GitHub repo
wget -O master.zip https://github.com/mongodb/mongodb-enterprise-kubernetes/archive/master.zip
unzip master.zip
# Initialize the helm and helm chart
helm init --upgrade
helm install mongodb-enterprise-kubernetes-master/helm_chart/ --name mongodb-enterprise

# If helm is not available
# Install the enterprise operator via YAML
kubectl apply -f mongodb-enterprise-kubernetes-master/mongodb-enterprise.yaml

Ops Manager configuration

• Make a note of the Ops Manager base URI
• Create a Project in Ops Manager
• Make a note of the Project ID
• Settings > Project Settings > General > Project ID
• Create a Public API key
• Account > Public API Access > API Keys
• Find your public IP address
• Whitelist your IP Address
• Account > Public API Access > API Whitelist
• Configure Version Manager with MongoDB versions of your interest Deployments > More > Version Manager

Make note of the Ops Manager Project ID and Create Public API Key & Whitelist your IP address

Update the scripts for your environment

Please edit the 
templates/environment.sh
 file with appropriate values for your environment. You must update the values for the properties before you start creating a replica set. The required set of properties are as shown below

OM_PROJECT_ID="<opsmanager_project_id>"
OM_USER_BASE64=$(echo "<opsmanager_userid>" | base64)
OM_API_KEY_BASE64=$(echo "<opsmanager_public_apikey>" | base64)
OM_URL="<opsmanager_uri>"
K8_NAMESPACE="<kubernetes_namespace>"
MONGODB_VERSION="<mongodb_version>"

For example, If you want to use MongoDB Cloud Manager and create a MonogDB v3.6.5 replica set then you must update the properies in the file 
templates/environment.sh
 as shown below.

OM_URL="https://cloud.mongodb.com/"
K8_NAMESPACE="mongodb-world"
MONGODB_VERSION="3.6.5"

Create a MongoDB replica set in Kubernetes

I have provided a sample template file for your convenience. Using the values defined in the environment file, 
templates/environment.sh
 and the template file, 
templates/generate-yaml-simple-replicaset.sh
, you could easily generate a new custom YAML file, 
samples/<kubernetes_namespace>-replicaset.yaml
. You would need to run the above YAML file via kubectl to create the MongoDB replica set.
Based on your download speed, the required resources/images are downloaded/created for you within a few seconds to a few minutes. When you run get all resources you would be see that there are 3 Pods, 1 Statefulset with Desired state of 3 and 2 services were created for you in the namespace you provided.

# display all the resources in the namespace
kubectl -n $K8_NAMESPACE get all
# NAME                             READY     STATUS    RESTARTS   AGE
# pod/mongodb-world-replicaset-0   1/1       Running   0          4m
# pod/mongodb-world-replicaset-1   1/1       Running   0          4m
# pod/mongodb-world-replicaset-2   1/1       Running   0          4m
# 
# NAME                                            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    #        AGE
# service/mongodb-world-replicaset-svc            ClusterIP   None            <none>        27017/TCP  #        4m
# service/mongodb-world-replicaset-svc-external   NodePort    10.43.252.153   <none>        # 27017:31750/TCP   4m
# 
# NAME                                        DESIRED   CURRENT   AGE

Finally, if you check the Ops Manager projects, then you would see a MongoDB deployment with the above Pods is created and the Automation and Monitoring is turned on as well.



Under the hood details of the helper script

Let’s try to understand what happens under the hood when you run the generated YAML. The 
generate-yaml-simple-replicaset.sh
 shell script creates a YAML file based on the environment variables you defined in the 
templates/environment.sh
 file. In my case, it generated a file called 
mongodb-world-replicaset.yaml
. It simplifies the overall process by performing below tasks
• Create a namespace in Kubernetes
• Create ConfigMap with the Ops Manager Project ID, baseUrl
• Create Secret with the base64 encoded values of Ops Manager Username and Public Api Key
• Create a MongoDbReplicaSet with 3 members, version: 3.6.5

# mongodb-world-replicaset.yaml contents
---
apiVersion: v1
kind: Namespace
metadata:
  name: "mongodb-world"
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: "mongodb-world-project"
  namespace: "mongodb-world"
data:
  projectId: "59*************6fed"
  baseUrl: "https://cloud.mongodb.com/"
---
apiVersion: v1
kind: Secret
metadata:
  name: "mongodb-world-credentials"
  namespace: "mongodb-world"
type: Opaque
data:
  user: "c2************UK"
  publicApiKey: "Nj******************zRiCg=="
---
apiVersion: mongodb.com/v1
kind: MongoDbReplicaSet
metadata:
  name: "mongodb-world-replicaset"
  namespace: "mongodb-world"
spec:
  members: 3
  version: "3.6.5"
  project: "mongodb-world-project"
  credentials: "mongodb-world-credentials"
  persistent: false  # For testing only
  podSpec:
    cpu: '0.25'
    memory: 512M
    storage: 2G

Connecting to MongoDB deployment

The connectivity to the MongoDB deployment within the Kubernetes cluster is very simple. Here in this example, I am connecting to the interactive terminal of first replica set member and using the mongo shell to connect to localhost.

kubectl -n ${K8_NAMESPACE}  exec -it ${K8_NAMESPACE}-replicaset-0 -- bin/bash
# mongodb@mongodb-world-replicaset-0:/$
/var/lib/mongodb-mms-automation/mongodb-linux-x86_64-3.6.5/bin/mongo
# MongoDB shell version v3.6.5
# connecting to: mongodb://mongodb-world-replicaset-0:27017/
# MongoDB server version: 3.6.5
# mongodb-world-replicaset:PRIMARY>

Since I am running these pods on the minikube, the Node IP address 10.128.0.7 is resolvable on my laptop. Therefore, I could connect to the MongoDB from outside of the cluster by using the Node Ports. However, you would need to use the Kubernetes ingress to allow connectivity from servers external to Kubernetes.

kubectl -n ${K8_NAMESPACE} describe pod/mongodb-world-replicaset-0 | grep 'Node:'
# Node:           10.128.0.7
kubectl -n ${K8_NAMESPACE} get services
# NAME                                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)           AGE
# mongodb-world-replicaset-svc            ClusterIP   None            <none>        27017/TCP         41m
# mongodb-world-replicaset-svc-external   NodePort    10.43.252.153   <none>        27017:31750/TCP   41m
mongo --host 10.128.0.7 --port 31750
# MongoDB shell version v3.6.5
# connecting to: mongodb://mongodb-world-replicaset-0:27017/
# MongoDB server version: 3.6.5

Remove the MongoDB Cluster

The below command will delete the MongoDbReplicaSet. The Ops Manager deployment in the given project should be removed as well. If for any reason this project still exists, you have to go to Deployment > … > Remove from Ops Manager to remove it completely.

kubectl delete -f samples/${K8_NAMESPACE}-replicaset.yaml

Check Enterprise Operator logs for errors

Sometimes you may notice that MongoDbReplicaSet creation did not succeed. To figure out what went wrong, you would have to check the logs on the mongodb-enterprise-operator pod.

# find the pod name for mongodb-enterprise-operator  using selectors
K8_OPERATOR_POD_NAME=$(kubectl -n mongodb get pods --selector=app=mongodb-enterprise-operator --output=jsonpath='{.items[0].metadata.name}')
# display the mongodb-enterprise-operator logs from mongodb namespace

Some of the typical errors, you may incur are related to
• Current IP Address is not added to Whitelist
• MongoDB Version not checked/binaries not available on Ops Manager
If you find such errors in your logs, then have to fix them first and recreate the operator pod before creating the replica set again. Since the operator is a deployment, you could simply delete the pod and a new one would be created for you.

# delete the existing pod after fixing the issue
kubectl -n mongodb delete pod $K8_OPERATOR_POD_NAME
sleep 5
# display all the resources in the namespace
kubectl -n ${K8_NAMESPACE} get all

Key Points & Other challenges

I am very positive that you are very excited if you got it working this far. Before you go ahead and deploy your first MongoDB deployment in your production Kubernetes, I want you to understand below key points.
• Pods are ephemeral. So make sure you use the persistent disks
• Make sure the Pods are provisioned across different nodes to ensure the High Availability
• Explicitly set the wiredTigerCacheSizeGB to 50% of Pod’s memory
• Use Affinity, Anti-Affinity to ensure Pods are distributed and resources are shared evenly

Summary

The support for deploying replica sets and clusters is just the beginning. The future release may add lot more features
• Scale up/down the number of members in the replica set
• Convert a replica set to sharded cluster
• Inject advanced mongod configurations, etc.
The MongoDB Enterprise Operator for Kubernetes brings in a lot of excitement for the developers in you. However, I want to stress an important point.

MongoDB offers redundancy and high availability of the database via replica sets. While the operator simplifies the deployment of MongoDB replica sets, you need to take proper measures while provisioning the Pods so that MongoDB is highly available across multiple Kubernetes Nodes and/or Data Centers. Most importantly the Pods should be allocated with the required hardware resources as per the sizing exercise.

Now that you understand all the crucial points for deploying MongoDB in Kubernetes, you might wonder, “How to size my deployments (find out how much hardware resources are needed)?”. Great question! But that’s a topic for another day.
Hopefully, you learned something new today on you scale the path to “Mastering MongoDB — One tip a day”.

Resources

• Introducing the MongoDB Enterprise Operator for Kubernetes and OpenShift — by Robert Walters
• GitHub repo: mongodb-enterprise-kubernetes
• Kubernetes Webinars by Janakiram

Tags