This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Getting Started

For guide to install, setup and manage MongoDB Operator

1 - Installation

MongoDB Operator installation, upgrade guide

MongoDB operator is based on the CRD framework of Kubernetes, for more information about the CRD framework please refer to the official documentation. In a nutshell, CRD is a feature through which we can develop our own custom API’s inside Kubernetes.

The API versions for MongoDB Operator available are:-

  • MongoDB
  • MongoDBCluster

MongoDB Operator requires a Kubernetes cluster of version >=1.16.0. If you have just started with the CRD and Operators, its highly recommended using the latest version of Kubernetes.

Setup of MongoDB operator can be easily done by using simple helm and kubectl commands.

Operator Setup by Helm

The setup can be done by using helm. The mongodb-operator can easily get installed using helm commands.

# Add the helm chart
$ helm repo add ot-helm https://ot-container-kit.github.io/helm-charts/
...
"ot-helm" has been added to your repositories

# Deploy the MongoDB Operator
$ helm install mongodb-operator ot-helm/mongodb-operator \
  --namespace ot-operators
...
Release "mongodb-operator" does not exist. Installing it now.
NAME: mongodb-operator
LAST DEPLOYED: Sun Jan  9 23:05:13 2022
NAMESPACE: ot-operators
STATUS: deployed
REVISION: 1

Once the helm chart is deployed, we can test the status of operator pod by using:

# Testing Operator
$ helm test mongodb-operator --namespace ot-operators
...
NAME:           mongodb-operator
LAST DEPLOYED:  Sun Jan  9 23:05:13 2022
NAMESPACE:      ot-operators
STATUS:         deployed
REVISION:       1
TEST SUITE:     mongodb-operator-test-connection
Last Started:   Sun Jan  9 23:05:54 2022
Last Completed: Sun Jan  9 23:06:01 2022
Phase:          Succeeded

Verify the deployment of MongoDB Operator using kubectl command.

# List the pod and status of mongodb-operator
$ kubectl get pods -n ot-operators -l name=mongodb-operator
...
NAME                               READY   STATUS    RESTARTS   AGE
mongodb-operator-fc88b45b5-8rmtj   1/1     Running   0          21d

Operator Setup by Kubectl

In any case using helm chart is not a possiblity, the MongoDB operator can be installed by kubectl commands as well.

As a first step, we need to setup a namespace and then deploy the CRD definitions inside Kubernetes.

# Setup of CRDs
$ kubectl create namespace ot-operators
$ kubectl apply -f https://github.com/OT-CONTAINER-KIT/mongodb-operator/raw/main/config/crd/bases/opstreelabs.in_mongodbs.yaml
$ kubectl apply -f https://github.com/OT-CONTAINER-KIT/mongodb-operator/raw/main/config/crd/bases/opstreelabs.in_mongodbclusters.yaml

Once we have namespace in the place, we need to setup the RBAC related stuff like:- ClusterRoleBindings, ClusterRole, Serviceaccount.

# Setup of RBAC account
$ kubectl apply -f https://raw.githubusercontent.com/OT-CONTAINER-KIT/mongodb-operator/main/config/rbac/service_account.yaml
$ kubectl apply -f https://raw.githubusercontent.com/OT-CONTAINER-KIT/mongodb-operator/main/config/rbac/role.yaml
$ kubectl apply -f https://github.com/OT-CONTAINER-KIT/mongodb-operator/blob/main/config/rbac/role_binding.yaml

As last part of the setup, now we can deploy the MongoDB Operator as deployment of Kubernetes.

# Deployment for MongoDB Operator
$ kubectl apply -f https://github.com/OT-CONTAINER-KIT/mongodb-operator/raw/main/config/manager/manager.yaml

Verify the deployment of MongoDB Operator using kubectl command.

# List the pod and status of mongodb-operator
$ kubectl get pods -n ot-operators -l name=mongodb-operator
...
NAME                               READY   STATUS    RESTARTS   AGE
mongodb-operator-fc88b45b5-8rmtj   1/1     Running   0          21d

2 - Standalone Setup

MongoDB database standalone setup guide

The MongoDB operator is capable for setting up MongoDB in the standalone mode with alot of additional power-ups like monitoring.

In this guide, we will see how we can set up MongoDB standalone with the help MongoDB operator and custom CRDS. We are going to use Kubernetes deployment tools like:- helm and kubectl.

Standalone architecture:

Setup using Helm Chart

Add the helm repository, so that MongoDB chart can be available for the installation. The repository can be added by:-

# Adding helm repository
$ helm repo add ot-helm https://ot-container-kit.github.io/helm-charts/
...
"ot-helm" has been added to your repositories

If the repository is added make sure you have updated it with the latest information.

# Updating ot-helm repository
$ helm repo update

Once all these things have completed, we can install MongoDB database by using:-

# Install the helm chart of MongoDB
$ helm install mongodb-ex --namespace ot-operators ot-helm/mongodb
...
NAME:          mongodb-ex
LAST DEPLOYED: Mon Jan 31 20:29:54 2022
NAMESPACE:     ot-operators
STATUS:        deployed
REVISION:      1
TEST SUITE:    None
NOTES:
  CHART NAME:    mongodb
  CHART VERSION: 0.1.0
  APP VERSION:   0.1.0

The helm chart for MongoDB standalone setup has been deployed.

Get the list of pods by executing:
    kubectl get pods --namespace ot-operators -l app=mongodb-ex-standalone

For getting the credential for admin user:
    kubectl get secrets -n ot-operators mongodb-ex-secret -o jsonpath="{.data.password}" | base64 -d

Verify the pod status and secret value by using:-

# Verify the status of the pods
$ kubectl get pods --namespace ot-operators -l app=mongodb-ex-standalone
...
NAME                      READY   STATUS    RESTARTS   AGE
mongodb-ex-standalone-0   2/2     Running   0          2m10s

# Verify the secret value
$ export PASSWORD=$(kubectl get secrets -n ot-operators mongodb-ex-secret -o jsonpath="{.data.password}" | base64 -d)
$ echo ${PASSWORD}
...
G7orFUuIrajGDK1iQzoD

Setup using Kubectl Commands

It is not a recommended way for setting for MongoDB database, it can be used for the POC and learning of MongoDB operator deployment.

All the kubectl related manifest are located inside the examples folder which can be applied using kubectl apply -f.

For an example:-

$ kubectl apply -f examples/basic/standalone.yaml -n ot-operators

Validation of MongoDB Database

To validate the state of MongoDB database, we can take the shell access of the MongoDB pod.

# For getting the MongoDB container
$ kubectl exec -it mongodb-ex-standalone-0 -c mongo -n ot-operators -- bash

Execute mongodb ping command to check the health of MongoDB.

# MongoDB ping command
$ mongosh --eval "db.adminCommand('ping')"
...
Current Mongosh Log ID:	61f81de07cacf368e9a25322
Connecting to:		mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000
Using MongoDB:		5.0.5
Using Mongosh:		1.1.7

{ ok: 1 }

We can also check the state of MongoDB by listing out the stats from admin database, but it would require username and password for same.

# MongoDB command for checking db stats
$ mongosh -u $MONGO_ROOT_USERNAME -p $MONGO_ROOT_PASSWORD --eval "db.stats()"
...
Current Mongosh Log ID:	61f820ab99bd3271e034d3b6
Connecting to:		mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000
Using MongoDB:		5.0.5
Using Mongosh:		1.1.7
{
  db: 'test',
  collections: 0,
  views: 0,
  objects: 0,
  avgObjSize: 0,
  dataSize: 0,
  storageSize: 0,
  totalSize: 0,
  indexes: 0,
  indexSize: 0,
  scaleFactor: 1,
  fileSize: 0,
  fsUsedSize: 0,
  fsTotalSize: 0,
  ok: 1
}

Create a database inside MongoDB database and try to insert some data inside it.

# create database inside MongoDB
$ use validatedb

$ db.user.insert({name: "Abhishek Dubey", age: 24})
$ db.user.insert({name: "Sajal Jain", age: 32})
...
WriteResult({ "nInserted" : 1 })

Let’s try to get out information from validatedb to see if write operation is successful or not.

$ db.user.find().pretty();
...
{
	"_id" : ObjectId("61fbe07d052a532fa6842a00"),
	"name" : "Abhishek Dubey",
	"age" : 24
}
{
	"_id" : ObjectId("61fbe1b7052a532fa6842a01"),
	"name" : "Sajal Jain",
	"age" : 32
}

3 - Replication Setup

MongoDB database replication setup guide

If we are running our application inside the production environment, in that case we should always go with HA architecture. MongoDB operator can also set up MongoDB database in the replication mode where there can be one primary instance and many secondary instances.

Architecture:-

In this guide, we will see how we can set up MongoDB replicated cluster using MongoDB operator and custom CRDS.

Setup using Helm Chart

Add the helm repository, so that MongoDB chart can be available for the installation. The repository can be added by:-

# Adding helm repository
$ helm repo add ot-helm https://ot-container-kit.github.io/helm-charts/
...
"ot-helm" has been added to your repositories

If the repository is added make sure you have updated it with the latest information.

# Updating ot-helm repository
$ helm repo update

Once all these things have completed, we can install MongoDB cluster database by using:-

# Installation of MongoDB replication cluster
$ helm install mongodb-ex-cluster --namespace ot-operators ot-helm/mongodb-cluster
...
NAME:          mongodb-ex-cluster
LAST DEPLOYED: Tue Feb  1 23:18:36 2022
NAMESPACE:     ot-operators
STATUS:        deployed
REVISION:      1
TEST SUITE:    None
NOTES:
  CHART NAME:    mongodb-cluster
  CHART VERSION: 0.1.0
  APP VERSION:   0.1.0

The helm chart for MongoDB standalone setup has been deployed.

Get the list of pods by executing:
    kubectl get pods --namespace ot-operators -l app=mongodb-ex-cluster-cluster

For getting the credential for admin user:
    kubectl get secrets -n ot-operators mongodb-ex-secret -o jsonpath="{.data.password}" | base64 -d

Verify the pod status of mongodb database cluster and secret value by using:-

# Verify the status of the mongodb cluster pods
$ kubectl get pods --namespace ot-operators -l app=mongodb-ex-cluster-cluster
...
NAME                           READY   STATUS    RESTARTS   AGE
mongodb-ex-cluster-cluster-0   2/2     Running   0          5m57s
mongodb-ex-cluster-cluster-1   2/2     Running   0          5m28s
mongodb-ex-cluster-cluster-2   2/2     Running   0          4m48s

# Verify the secret value
$ export PASSWORD=$(kubectl get secrets -n ot-operators mongodb-ex-cluster-secret -o jsonpath="{.data.password}" | base64 -d)
$ echo ${PASSWORD}
...
fEr9FScI9ojh6LSh2meK

Setup using Kubectl Commands

It is not a recommended way for setting for MongoDB database, it can be used for the POC and learning of MongoDB operator deployment.

All the kubectl related manifest are located inside the examples folder which can be applied using kubectl apply -f.

For an example:-

$ kubectl apply -f examples/basic/clusterd.yaml -n ot-operators

Validation of MongoDB Cluster

Once the cluster is created and in running state, we should verify the health of the MongoDB cluster.

# Verifying the health of the cluster
$ kubectl exec -it mongodb-ex-cluster-cluster-0 -n ot-operators -- bash

$ mongo -u $MONGO_ROOT_USERNAME -p $MONGO_ROOT_PASSWORD --eval "db.adminCommand( { replSetGetStatus: 1 } )"
...
{
	"set" : "mongodb-ex-cluster",
	"date" : ISODate("2022-02-03T08:52:09.257Z"),
	"myState" : 1,
	"term" : NumberLong(1),
	"syncSourceHost" : "",
	"syncSourceId" : -1,
	"heartbeatIntervalMillis" : NumberLong(2000),
	"majorityVoteCount" : 2,
	"writeMajorityCount" : 2,
	"votingMembersCount" : 3,
	"writableVotingMembersCount" : 3,
	"optimes" : {
		"lastCommittedOpTime" : {
			"ts" : Timestamp(1643878322, 1),
			"t" : NumberLong(1)
		},
		"lastCommittedWallTime" : ISODate("2022-02-03T08:52:02.111Z"),
		"readConcernMajorityOpTime" : {
			"ts" : Timestamp(1643878322, 1),
			"t" : NumberLong(1)
		},
		"appliedOpTime" : {
			"ts" : Timestamp(1643878322, 1),
			"t" : NumberLong(1)
		},
		"durableOpTime" : {
			"ts" : Timestamp(1643878322, 1),
			"t" : NumberLong(1)
		},
		"lastAppliedWallTime" : ISODate("2022-02-03T08:52:02.111Z"),
		"lastDurableWallTime" : ISODate("2022-02-03T08:52:02.111Z")
	},
	"lastStableRecoveryTimestamp" : Timestamp(1643878312, 1),
	"electionCandidateMetrics" : {
		"lastElectionReason" : "electionTimeout",
		"lastElectionDate" : ISODate("2022-02-01T17:50:49.975Z"),
		"electionTerm" : NumberLong(1),
		"lastCommittedOpTimeAtElection" : {
			"ts" : Timestamp(1643737839, 1),
			"t" : NumberLong(-1)
		},
		"lastSeenOpTimeAtElection" : {
			"ts" : Timestamp(1643737839, 1),
			"t" : NumberLong(-1)
		},
		"numVotesNeeded" : 2,
		"priorityAtElection" : 1,
		"electionTimeoutMillis" : NumberLong(10000),
		"numCatchUpOps" : NumberLong(0),
		"newTermStartDate" : ISODate("2022-02-01T17:50:50.072Z"),
		"wMajorityWriteAvailabilityDate" : ISODate("2022-02-01T17:50:50.926Z")
	},
	"members" : [
		{
			"_id" : 0,
			"name" : "mongodb-ex-cluster-cluster-0.mongodb-ex-cluster-cluster.ot-operators:27017",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 140603,
			"optime" : {
				"ts" : Timestamp(1643878322, 1),
				"t" : NumberLong(1)
			},
			"optimeDate" : ISODate("2022-02-03T08:52:02Z"),
			"lastAppliedWallTime" : ISODate("2022-02-03T08:52:02.111Z"),
			"lastDurableWallTime" : ISODate("2022-02-03T08:52:02.111Z"),
			"syncSourceHost" : "",
			"syncSourceId" : -1,
			"infoMessage" : "",
			"electionTime" : Timestamp(1643737849, 1),
			"electionDate" : ISODate("2022-02-01T17:50:49Z"),
			"configVersion" : 1,
			"configTerm" : 1,
			"self" : true,
			"lastHeartbeatMessage" : ""
		},
		{
			"_id" : 1,
			"name" : "mongodb-ex-cluster-cluster-1.mongodb-ex-cluster-cluster.ot-operators:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 140490,
			"optime" : {
				"ts" : Timestamp(1643878322, 1),
				"t" : NumberLong(1)
			},
			"optimeDurable" : {
				"ts" : Timestamp(1643878322, 1),
				"t" : NumberLong(1)
			},
			"optimeDate" : ISODate("2022-02-03T08:52:02Z"),
			"optimeDurableDate" : ISODate("2022-02-03T08:52:02Z"),
			"lastAppliedWallTime" : ISODate("2022-02-03T08:52:02.111Z"),
			"lastDurableWallTime" : ISODate("2022-02-03T08:52:02.111Z"),
			"lastHeartbeat" : ISODate("2022-02-03T08:52:08.021Z"),
			"lastHeartbeatRecv" : ISODate("2022-02-03T08:52:07.926Z"),
			"pingMs" : NumberLong(65),
			"lastHeartbeatMessage" : "",
			"syncSourceHost" : "mongodb-ex-cluster-cluster-2.mongodb-ex-cluster-cluster.ot-operators:27017",
			"syncSourceId" : 2,
			"infoMessage" : "",
			"configVersion" : 1,
			"configTerm" : 1
		},
		{
			"_id" : 2,
			"name" : "mongodb-ex-cluster-cluster-2.mongodb-ex-cluster-cluster.ot-operators:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 140490,
			"optime" : {
				"ts" : Timestamp(1643878322, 1),
				"t" : NumberLong(1)
			},
			"optimeDurable" : {
				"ts" : Timestamp(1643878322, 1),
				"t" : NumberLong(1)
			},
			"optimeDate" : ISODate("2022-02-03T08:52:02Z"),
			"optimeDurableDate" : ISODate("2022-02-03T08:52:02Z"),
			"lastAppliedWallTime" : ISODate("2022-02-03T08:52:02.111Z"),
			"lastDurableWallTime" : ISODate("2022-02-03T08:52:02.111Z"),
			"lastHeartbeat" : ISODate("2022-02-03T08:52:09.022Z"),
			"lastHeartbeatRecv" : ISODate("2022-02-03T08:52:07.475Z"),
			"pingMs" : NumberLong(1),
			"lastHeartbeatMessage" : "",
			"syncSourceHost" : "mongodb-ex-cluster-cluster-0.mongodb-ex-cluster-cluster.ot-operators:27017",
			"syncSourceId" : 0,
			"infoMessage" : "",
			"configVersion" : 1,
			"configTerm" : 1
		}
	],
	"ok" : 1,
	"$clusterTime" : {
		"clusterTime" : Timestamp(1643878322, 1),
		"signature" : {
			"hash" : BinData(0,"M4Mj+BFC8//nI2QIT+Hic/N6N4g="),
			"keyId" : NumberLong("7059800308947353604")
		}
	},
	"operationTime" : Timestamp(1643878322, 1)
}

Create a database inside MongoDB database and try to insert some data inside it.

# create database inside MongoDB
$ use validatedb

$ db.user.insert({name: "Abhishek Dubey", age: 24})
$ db.user.insert({name: "Sajal Jain", age: 32})
...
WriteResult({ "nInserted" : 1 })

Let’s try to get out information from validatedb to see if write operation is successful or not.

# Get inside the secondary pod
$ kubectl exec -it mongodb-ex-cluster-cluster-1 -n ot-operators -- bash

# Login inside the MongoDB shell
$ mongo -u $MONGO_ROOT_USERNAME -p $MONGO_ROOT_PASSWORD

List out the collection data on MongoDB secondary node by using mongo shell commands.

$ secondaryOk()
$ db.user.find().pretty();
...
{
	"_id" : ObjectId("61fbe23d5b403e9fb63bc374"),
	"name" : "Abhishek Dubey",
	"age" : 24
}
{
	"_id" : ObjectId("61fbe2465b403e9fb63bc375"),
	"name" : "Sajal Jain",
	"age" : 32
}

As we can see, we are able to list out the data which have been written inside the primary node is also queryable from the secondary nodes as well.

4 - Failover Testing

Failover testing for the MongoDB replicated setup

In this section, we will deactivate/delete the specific nodes (mainly primary node) to force the replica set to make an election and select a new primary node.

Before deleting/deactivating the primary node, we will try to put some dummy data inside it.

# Get inside the primary node pod
$ kubectl exec -it mongodb-ex-cluster-cluster-0 -n ot-operators -- bash

# Login inside the MongoDB shell
$ mongo -u $MONGO_ROOT_USERNAME -p $MONGO_ROOT_PASSWORD

Once we are inside the MongoDB primary pod, we will create a db called failtestdb and create a collection inside it with some dummy data.

# Creation of the MongoDB database
$ use failtestdb

# Collection creation inside MongoDB database
$ db.user.insert({name: "Abhishek Dubey", age: 24})
...
WriteResult({ "nInserted" : 1 })

Let’s check if the data is written properly or not inside the primary node.

# Find the data available inside the collection
$ db.user.find().pretty();
...
{
	"_id" : ObjectId("61fc0e19a63b3bfa4f30d5e2"),
	"name" : "Abhishek Dubey",
	"age" : 24
}

Deactivating/Deleting MongoDB Primary and Secondary

Once we have established that the database and content is written properly inside the MongoDB database. Now let’s try to delete the primary pod to see what is the impact of it.

# Delete the primary pod
$ kubectl delete pod mongodb-ex-cluster-cluster-0 -n ot-operators
...
pod "mongodb-ex-cluster-cluster-0" deleted

When the pod is up and running, get inside the same pod and see the role assigned to that MongoDB node. Also, the replication status of the cluster.

# Login again inside the primary pod
$ kubectl exec -it mongodb-ex-cluster-cluster-0 -n ot-operators -- bash

# Login inside the MongoDB shell
$ mongo -u $MONGO_ROOT_USERNAME -p $MONGO_ROOT_PASSWORD

# Check the mongo node is still primary or not
$ db.hello()
...
{
	"topologyVersion" : {
		"processId" : ObjectId("61fc125bbb845ecebf0b6c1e"),
		"counter" : NumberLong(4)
	},
	"hosts" : [
		"mongodb-ex-cluster-cluster-0.mongodb-ex-cluster-cluster.ot-operators:27017",
		"mongodb-ex-cluster-cluster-1.mongodb-ex-cluster-cluster.ot-operators:27017",
		"mongodb-ex-cluster-cluster-2.mongodb-ex-cluster-cluster.ot-operators:27017"
	],
	"setName" : "mongodb-ex-cluster",
	"setVersion" : 1,
	"isWritablePrimary" : false,
	"secondary" : true,
	"primary" : "mongodb-ex-cluster-cluster-1.mongodb-ex-cluster-cluster.ot-operators:27017",
	}
}

As we can see in the above command, output this pod has become the secondary pod and mongodb-ex-cluster-cluster-1 has become the primary pod of MongoDB. Also, this pod is now non-writable state. Let’s check if we can fetch out the collection data from MongoDB database.

# Change the database to failtestdb
$ use failtestdb

# Set the slave read flag of MongoDB
$ secondaryOk()

# Find the data available inside the collection
$ db.user.find().pretty();
...
{
	"_id" : ObjectId("61fc0e19a63b3bfa4f30d5e2"),
	"name" : "Abhishek Dubey",
	"age" : 24
}