Deep Thought

Exercise_MongoDB_ReplicaSet

4 min read

Exercise: MongoDB Replica Set — CAP Live

Belongs to: Document_Databases · Duration: ~50 min · Work in pairs

You know MongoDB as a single server. Today you run it as a 3-node cluster and break it on purpose: network partition, primary crash, loss of the majority. You will see with your own eyes whether MongoDB really behaves CP.

You need three terminal windows:

WindowName in this sheetUsed for
1HOSTdocker commands in the nosql-lab folder
2M1MongoDB shell on mongo1
3M3MongoDB shell on mongo3

Before each step, write down what you expect. Then write down what you observe.

0. Start the cluster (5 min)

HOST:

docker compose --profile mongo up -d
docker exec -it mongo1 mongosh

That window is now M1. Create the replica set:

rs.initiate({
  _id: "rs0",
  members: [
    { _id: 0, host: "mongo1:27017" },
    { _id: 1, host: "mongo2:27017" },
    { _id: 2, host: "mongo3:27017" }
  ]
})

Wait ~10 seconds and press Enter a few times. The prompt changes to rs0 [direct: primary] test>.

Check the roles — you will need this command often:

rs.status().members.map(m => m.name + "  " + m.stateStr + "  health=" + m.health)

1. Write on the primary, read on a secondary (8 min)

M1:

use shop
db.products.insertOne({ _id: 1, name: "Keyboard", price: 40 })

Open a new terminal and connect to mongo3. That window is now M3:

docker exec -it mongo3 mongosh

M3:

use shop
db.getMongo().setReadPref("secondaryPreferred")
db.products.find()                                          // → ?
db.products.insertOne({ _id: 99, name: "Hack", price: 1 })  // → ?

Questions: Is the keyboard on mongo3? Why can’t you insert there?

2. Network partition (15 min)

Now we cut mongo3’s network cable — the container keeps running, but can’t reach the other nodes.

HOST:

docker network disconnect nosql-net mongo3

Wait ~15 seconds. M1: check the roles with the rs.status() command from step 0. → ?

M1: change the price (default write concern: majority)

db.products.updateOne({ _id: 1 }, { $set: { price: 45 } })    // → works?

M3:

db.products.find()                                          // → price?

M1: now insist that all three nodes confirm the write, but wait at most 5 seconds:

db.products.insertOne(
  { _id: 2, name: "Mouse", price: 25 },
  { writeConcern: { w: 3, wtimeout: 5000 } }
)                                                           // → ?
db.products.find()                                          // → is the mouse there?

Look closely at the last two results. Was the write rejected — or just not confirmed?

Reconnect the cable. HOST:

docker network connect --alias mongo3 nosql-net mongo3

Wait ~10 seconds. M3: db.products.find() → ?

Questions: a) During the partition, mongo3 still answered reads. Was the answer correct? Which letter of CAP did mongo3 give up for the reads? b) Why did the w: "majority" write work, but the w: 3 write time out? c) What does wtimeout not do with the data?

3. The primary crashes (10 min)

HOST:

docker stop mongo1

M1 is now dead — close it. M3: run the rs.status() command every few seconds until a new primary appears. How long did the election take? Which node won?

Write a new product on the new primary (if the new primary is mongo2, open a shell there with docker exec -it mongo2 mongosh):

use shop
db.products.insertOne({ _id: 3, name: "Webcam", price: 60 })

Bring the old primary back. HOST:

docker start mongo1

Open a new shell on mongo1 (docker exec -it mongo1 mongosh) and check the roles again. Is mongo1 primary again? Does it have the webcam?

use shop
db.getMongo().setReadPref("secondaryPreferred")
db.products.find()

Close this shell with exit.

4. Losing the majority (7 min)

Stop two of the three nodes. HOST:

docker stop mongo1 mongo2

M3 (reconnect with docker exec -it mongo3 mongosh if your shell was closed), wait ~15 seconds:

rs.status().members.map(m => m.name + "  " + m.stateStr + "  health=" + m.health)
use shop
db.getMongo().setReadPref("secondaryPreferred")
db.products.find()                                          // → ?
db.products.insertOne({ _id: 4, name: "Microphone", price: 110 })   // → ?

Questions: Is there still a primary? Can you read? Can you write? Why does a single node not just promote itself?

5. Clean up

HOST:

docker compose --profile mongo down

Hand-in

  1. Your expected vs. observed results for steps 1–4.
  2. Answers to the questions in steps 1–4.
  3. Complete this sentence with arguments from today: “By default MongoDB is a CP system, because … — but it can behave like AP when …”
  4. Fill in the document column of the classification grid.

Graph View

Backlinks